Signal encoding for difficult environments

ABSTRACT

This disclosure relates to advanced image signal processing technology including encoded signals and digital watermarking. We disclose methods, systems and apparatus for selecting which ink(s) should be selected to carry an encoded signal for a given machine-vision wavelength for a retail package or other printed design. We also disclose retail product packages and other printed objects, and methods to generate such, including a sparse mark in a first ink and an overprinted ink flood in a second ink. The first ink and the second ink are related through tack and spectral reflectance difference. Of course, other methods, packages, objects, systems and apparatus are described in this disclosure.

RELATED APPLICATION DATA

This application is a Continuation of U.S. application Ser. No.15/418,364, filed Jan. 27, 2017 (U.S. Pat. No. 10,255,649), which claimsthe benefit of US Provisional Patent Application Nos. 62/430,297, filedDec. 5, 2016, 62/375,418, filed Aug. 15, 2016 and 62/377,419, filed Aug.19, 2016. The 15/418,364 application is also a Continuation in Part ofU.S. patent application Ser. No. 15/261,005, filed Sep. 9, 2016(published as US 2018-0047126 A1), which claims the benefit of USProvisional Patent Application Nos. 62/375,418, filed Aug. 15, 2016 and62/377,419, filed Aug. 19, 2016. This application is also related toU.S. patent application Ser. No. 14/616,686, filed Feb. 7, 2015 (U.S.Pat. No. 9,380,186), 14/725,399, filed May 29, 2015 (published as US2016-0275639 A1), 15/072,884, filed Mar. 17, 2016 (issued as U.S. Pat.No. 10,424,038), 14/588,636, filed Jan. 2, 2015 (published as US2015-0187039 A1, issued as US Patent No. No. 9,401,001), and 15/137,401,filed Apr. 25, 2016 (published as US 2016-0316098 A1; issued as U.S.Pat. No. 9,565,335). Each of the patent documents mentioned above ishereby incorporated herein by reference in its entirety, including alldrawings and any appendices

TECHNICAL FIELD

This disclosure relates to advanced signal processing technologyincluding image processing and encoded signaling techniques such asembedding and digital watermarking.

BACKGROUND AND SUMMARY

Portions of this disclosure are described in terms of, e.g., encodedsignals for digital designs, product packaging (sometimes just referredto herein as “packaging” or “package”) and other objects, e.g.,including printed objects such as tags, product and/or clothing hangtags, labels, product literature, etc. These encoding techniques can beused, e.g., to alter or transform how color inks are printed on variousphysical substrates. The alterations or transformations preferablyresult in a printed design carrying machine readable indicia on asurface of a physical object.

Various forms of signal encoding (or “embedding”) include, e.g.,“steganographic encoding” and “digital watermarking.” Digitalwatermarking is a process for transforming physical or electronic mediato embed a machine-readable code (or “auxiliary data” or “informationsignal”) into the media. In some cases the media is transformed suchthat the embedded code is “obscured” or “generally imperceptible”relative to an overt symbology (e.g., 1D or 2D barcode), yet may bedetected through an automated detection process. Obscured and generallyimperceptible in this context means that the luminance/chrominancevariations in the artwork due to the digital watermarking are notnoticeable to a human viewer inspecting the package from a usualdistance (e.g., 20 inches) under normal retail lighting (e.g., 50-85foot candles), who has not previously been alerted to the existence ofthe digital watermarking.

Digital watermarking is often applied to electronic or physical objectssuch as printed objects, images, audio signals, and video signals.However, it may also be applied to other types of objects, including,e.g., product packaging, electronics such as circuit boards and CPUs,stickers, logos, product hang tags, line-art, software,multi-dimensional graphics models, and surface textures of such objects.

In this document we use the terms “digital watermark” and “watermark”(and various forms thereof) interchangeably.

Auxiliary data embedding systems typically include two components: anencoder (or embedder) that embeds the auxiliary signal in a host imageor object, and a decoder (or detector) that detects and reads theembedded auxiliary signal from the host image or object. The encoder mayembed the auxiliary signal by altering or transforming a host image orobject to carry the auxiliary data. The detection component analyzes asuspect image, object or signal to detect whether an auxiliary signal ispresent, and if so, extracts or reads information carried in it.

Several particular digital watermarking and auxiliary data embeddingtechniques have been developed. The reader is presumed to be familiarwith the literature in this field. Particular techniques for embeddingand detecting imperceptible digital watermarks are detailed in theassignee's patent documents including US Published Patent ApplicationNos. 20150156369 and 20160217547; U.S. patent application Ser. No.14/725,399, filed May 29, 2015 (issued as U.S. Pat. No. 9,635,378), and14/842,575, filed Sep. 1, 2015 (issued as U.S. Pat. No. 9,819,950);International Application No. PCT/US2015/44904, filed Aug. 12, 2015(published as WO 2016025631 Al) and U.S. Pat. Nos. 7,054,461, 7,286,685,and 9,129,277. Related technology is detailed in Assignee's U.S. patentapplication Ser. No. 15/073,483, filed Mar. 17, 2016 (issued as

U.S. Pat. No. 9,754,341). Each of the patent documents mentioned in thisparagraph are hereby incorporated herein by reference in its entirety,including all drawings and any appendices.

One aspect of the disclosure is an image processing method comprising:obtaining first color values representing luminance, color channel ‘a’and color channel ‘b’ for a first color; obtaining second color valuesrepresenting luminance, color channel ‘a’ and color channel ‘b’ for asubstrate or a background color; obtaining first reflectance values forthe first color at a machine-vision wavelength; obtaining secondreflectance values for the substrate or the background color at themachine-vision wavelength; combining the first reflectance values andthe second reflectance values to yield a reflectance difference; usingone or more programmed processors, determining an encoded signal errorthat is associated with the first color values, the second color valuesand the reflectance difference; using one or more programmed processors,determining a color error that is associated with the first colorvalues, the second color values and the reflectance difference; andcombining the encoded signal error and the color error to yield acombined error, and evaluating the combined error to determine whetherto transform digital imagery to carry an encoded signal.

Another aspect of the disclosure is an image processing methodcomprising: obtaining first color values representing luminance*, colorchannel ‘a*’ and color channel ‘b*’ for a first plurality of colors;obtaining second color values representing luminance*, color channel‘a*’ and color channel ‘b*’ for a substrate or a background color; usingone or more programmed processors, determining an encoded signal errorthat is associated with the first color values, the second color values,and a first machine-vision wavelength difference between the first colorand the substrate or the background color; using one or more programmedprocessors, determining a color error that is associated with the firstcolor values, the second color values, and the first machine-visionwavelength difference between the first color and the substrate or thebackground color; and combining the encoded signal error and the colorerror for each of the first plurality of colors to yield a plurality ofcombined errors, and evaluating the combined errors to determine atarget color from the plurality of colors in terms of signal robustnessand signal visibility associated with the first machine-visionwavelength.

Yet another aspect of the disclosure is a retail product packagecomprising: a first substrate comprising a first area; a sparse markpattern printed within the first area with a first ink; an ink floodprinted over the sparse mark pattern and the first substrate within thefirst area with a second ink, in which the second ink comprises a largertack or adhesion with the first substrate relative to a tack or adhesionwith the first ink, the first area comprising a first region comprisinga layer of ink flood and a layer of first substrate, the first areafurther comprising a second region comprising a layer of ink flood, alayer of sparse mark pattern and a layer of first substrate, in whichthe first region and the second region comprise a spectral reflectancedifference at a machine-vision wavelength in the range of 8%-60%.

Further aspects, features and advantages will become even more apparentwith reference to the following detailed description, claims andaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a block diagram of a signal encoder for encoding a digitalpayload signal into an image signal.

FIG. 2 is a block diagram of a compatible signal decoder for extractingthe digital payload signal from an image signal.

FIG. 3 is a flow diagram illustrating operations of a signal generator.

FIG. 4 is a diagram illustrating embedding of an auxiliary signal intohost image signal.

FIG. 5 is a flow diagram illustrating a method for decoding a payloadsignal from a host image signal.

FIG. 6 is a diagram illustrating an example of a sparse signalgenerator.

FIG. 7 is a diagram illustrating a refinement of a sparse signalgenerator like the one in FIG. 6.

FIG. 8 is a histogram of a digital watermark signal component.

FIG. 9 is a histogram of another digital watermark signal component.

FIG. 10 is a histogram of a combination of the digital watermark signalcomponents of FIGS. 8 and 9, and also depicts examples of differentthresholds used to generate a binary image comprising black and whitepixels from an image comprised of multi-valued pixels.

FIG. 11 is a diagram illustrating another refinement of the sparsesignal generator of FIG. 6.

FIG. 12 is a diagram illustrating application of a threshold to awatermark signal, and the resulting output for three differentthresholds.

FIG. 13 illustrates a portion of a sparse signal.

FIG. 14 illustrates the sparse signal of FIG. 13, modified to reduce thesignal using a line screen approach.

FIG. 15A shows various colors along with their respective “DigimarcBarcode” (also referred to as “DB”) scores.

FIG. 15B shows reflectance curves for a Pantone 9520 C and Paper WhiteC.

FIG. 15C shows reflectance curves for Paper White C, Pantone 9520 C andan “ideal” (but hypothetical) embedding color.

FIG. 15D shows a Contrast Sensitivity Function (CSF) for the Human Eye.

FIG. 16 shows two printed areas including encoded signals.

FIG. 17 shows reflectance curves for a Pantone Blue 0821 C and Pantone333 C.

FIG. 18A shows a printed patch including black printed first, withPantone 9520 C printed over the top of the black.

FIG. 18B shows a printed patch including Pantone 9520 C printed first,with black printed on top.

FIG. 19 illustrates an electronic device in which ink selection,encoding and decoding may be implemented.

FIG. 20A is a graph showing reflectance differences across a colorspectrum.

FIG. 20B shows an encoded patch, and white and Cyan patches.

FIG. 21A is a graph showing reflectance differences for differentproduct fill colors across a color spectrum.

FIG. 21B shows two patches, one for a clear substrate with opaque whiteink and white fill (left) and another for a clear substrate with opaquewhite ink and black fill (right).

FIG. 22A is a graph showing reflectance differences for differentproduct fill colors vs. Cyan across a color spectrum.

FIG. 22B shows two patches, one for a clear substrate with opaque whiteink and white fill (left, with resulting colors for the white and cyan)and another for a clear substrate with opaque white ink and black fill(right, with resulting colors for the white and cyan).

FIGS. 23A-23D collective represent a quality ruler increasing in inkamount percentage from FIG. 23A (slight) to FIG. 23D (strong).

FIG. 24 includes 10 test sparse mark patch samples, with a solid versionof each color along with a corresponding Pantone number.

FIG. 25 shows an expanded example of one of the sparse mark patchesshown in FIG. 24.

FIG. 26A shows a plotting of mean subjective visibility testing of thesparse mark patch samples shown in FIG. 24.

FIG. 26B show a correlation between the mean subjective visibilitytesting of FIG. 26A and an objective DB Score.

DETAILED DESCRIPTION Introduction

The following detailed description is divided into four (4) generalsections. It should be understood from the outset, however, that weexpressly contemplate combining subject matter from one such sectionwith one or more of the other sections. Thus, the sections and sectionheadings are provided for the reader's convenience and are not intendedto impose restrictions or limitations. The sections include: I. SignalEncoder and Decoder; II. Sparse Marks; III. Color Selection and InkTrapping; and IV. Operating Environments.

I. Signal Encoder and Decoder

Encoder/Decoder

FIG. 1 is a block diagram of a signal encoder for encoding a digitalpayload signal into an image signal. FIG. 2 is a block diagram of acompatible signal decoder for extracting the digital payload signal froman image signal.

While the signal encoder and decoder may be used for communicating adata channel for many applications, one objective for use in physicalobjects is robust signal communication through images formed on andcaptured from these objects. Signal encoders and decoders, like those inthe Digimarc Barcode Platform from Digimarc Corporation, communicateauxiliary data in a data carrier within image content. Encoding anddecoding is applied digitally, yet the signal survives digital to analogtransformation and analog to digital transformation. For example, theencoder generates a modulated digital image that is converted to arendered form, such as a printed image. The modulated digital imageincludes the encoded signal prior to rendering. Prior to decoding, areceiving device has or communicates with an imager to capture themodulated signal, convert it to an electric signal, which is digitizedand then processed by the FIG. 2 signal decoder.

Inputs to the signal encoder include a host image 220 and auxiliary datapayload 222. The objectives of the encoder include encoding a robustsignal with desired payload capacity per unit of host signal (e.g., aunit may include the spatial area of a two-dimensional tile within thehost signal), while maintaining perceptual quality. In some cases, theremay be very little variability or presence of a host signal. In thiscase, there is little host interference on the one hand, yet little hostcontent in which to mask the presence of the data channel within animage. Some examples include a package design that is devoid of muchimage variability (e.g., a single, uniform color). See, e.g., Ser. No.14/725,399, entitled SPARSE MODULATION FOR ROBUST SIGNALING ANDSYNCHRONIZATION, incorporated herein by reference in its entirety.

The auxiliary data payload 222 includes the variable data information tobe conveyed in the data channel, possibly along with other protocol dataused to facilitate the communication. The protocol of the auxiliary dataencoding scheme comprises the format of the auxiliary data payload,error correction coding schemes, payload modulation methods (such as thecarrier signal, spreading sequence, encoded payload scrambling orencryption key), signal structure (including mapping of modulated signalto embedding locations within a tile), error detection in payload (CRC,checksum, etc.), perceptual masking method, host signal insertionfunction (e.g., how auxiliary data signal is embedded in or otherwisecombined with host image signal in a package or label design), and/orsynchronization method and signals.

The protocol defines the manner in which the signal is structured andencoded for robustness, perceptual quality and/or data capacity. For aparticular application, there may be a single protocol, or more than oneprotocol, depending on application requirements. Examples of multipleprotocols include cases where there are different versions of thechannel, different channel types (e.g., several digital watermark layerswithin a host). Different versions may employ different robustnessencoding techniques or different data capacity. Protocol selector module224 determines the protocol to be used by the encoder for generating adata signal. It may be programmed to employ a particular protocoldepending on the input variables, such as user control, applicationspecific parameters, or derivation based on analysis of the host signal.

Perceptual analyzer module 226 analyzes the input host signal todetermine parameters for controlling signal generation and embedding, asappropriate. It is not necessary in certain applications, while inothers it may be used to select a protocol and/or modify signalgeneration and embedding operations. For example, when encoding in hostcolor images that will be printed or displayed, the perceptual analyzer256 is used to ascertain color content and masking capability of thehost image. The output of this analysis, along with the rendering method(display or printing device) and rendered output form (e.g., ink andsubstrate) is used to control auxiliary signal encoding in particularcolor channels (e.g., one or more channels of process inks, Cyan,Magenta, Yellow, or Black (CMYK) or spot colors), perceptual models, andsignal protocols to be used with those channels. Please see, e.g., ourwork on visibility and color models used in perceptual analysis in ourU.S. application Ser. No. 14/616,686 (issued as U.S. Pat. No.9,380,186), 14/588,636 (published as US 2015-0187039 A1) and 13/975,919(issued as U.S. Pat. No. 9,449,357), Patent Application Publication No.US 2010-0150434 A1, and U.S. Pat. No. 7,352,878, which are herebyincorporated by reference in their entirety.

The perceptual analyzer module 226 also computes a perceptual model, asappropriate, to be used in controlling the modulation of a data signalonto a data channel within image content as described below.

The signal generator module 228 operates on the auxiliary data andgenerates a data signal according to the protocol. It may also employinformation derived from the host signal, such as that provided byperceptual analyzer module 226, to generate the signal. For example, theselection of data code signal and pattern, the modulation function, andthe amount of signal to apply at a given embedding location may beadapted depending on the perceptual analysis, and in particular on theperceptual model and perceptual mask that it generates. Please see belowand the incorporated patent documents for additional aspects of thisprocess.

Embedder module 230 takes the data signal and modulates it into an imageby combining it with the host image. The operation of combining may bean entirely digital signal processing operation, such as where the datasignal modulates the host signal digitally, may be a mixed digital andanalog process or may be purely an analog process (e.g., where renderedoutput images, with some signals being modulated data and others beinghost image content, such as the various layers of a package designfile).

There are a variety of different functions for combining the data andhost in digital operations. One approach is to adjust the host signalvalue as a function of the corresponding data signal value at anembedding location, which is limited or controlled according to theperceptual model and a robustness model for that embedding location. Theadjustment may be altering the host image by adding a scaled data signalor multiplying by a scale factor dictated by the data signal valuecorresponding to the embedding location, with weights or thresholds seton the amount of the adjustment according to the perceptual model,robustness model, and/or available dynamic range. The adjustment mayalso be altering by setting the modulated host signal to a particularlevel (e.g., quantization level) or moving it within a range or bin ofallowable values that satisfy a perceptual quality or robustnessconstraint for the encoded data.

As detailed further below, the signal generator 228 produces a datasignal with data elements that are mapped to embedding locations in animage tile. These data elements are modulated onto the host image at theembedding locations. A tile may include a pattern of embeddinglocations. The tile derives its name from the way in which it isrepeated in contiguous blocks of a host signal, but it need not bearranged this way. In image-based encoders, we may use tiles in the formof a two dimensional array (e.g., 128×128, 256×256, 512×512) ofembedding locations. The embedding locations correspond to host signalsamples at which an encoded signal element is embedded in an embeddingdomain, such as a spatial domain (e.g., pixels at a spatial resolution),frequency domain (frequency components at a frequency resolution), orsome other feature space. We sometimes refer to an embedding location asa bit cell, referring to a unit of data (e.g., an encoded bit or chipelement) encoded within a host signal at the location of the cell. Againplease see the documents incorporated herein for more information onvariations for particular type of media.

The operation of combining may include one or more iterations ofadjustments to optimize the modulated host for perceptual quality orrobustness constraints. One approach, for example, is to modulate thehost image so that it satisfies a perceptual quality metric asdetermined by perceptual model (e.g., visibility model) for embeddinglocations across the signal. Another approach is to modulate the hostimage so that it satisfies a robustness metric across the signal. Yetanother is to modulate the host image according to both the robustnessmetric and perceptual quality metric derived for each embeddinglocation. The incorporated documents provide examples of thesetechniques. Below, we highlight a few examples. See, e.g., U.S. patentapplication Ser. No. 13/975,919; and see also, U.S. patent applicationSer. No. 14/588,636, filed Jan. 2, 2015 (published as US 2015-0187039A1), filed Jan. 2, 2015, and Ser. No. 15/137,401, filed Apr. 25, 2016(published as US 2016-0316098 A1), which are each hereby incorporated byreference in its entirety.

For color images, the perceptual analyzer generates a perceptual modelthat evaluates visibility of an adjustment to the host by the embedderand sets levels of controls to govern the adjustment (e.g., levels ofadjustment per color direction, and per masking region). This mayinclude evaluating the visibility of adjustments of the color at anembedding location (e.g., units of noticeable perceptual difference incolor direction in terms of CIE Lab values), Contrast SensitivityFunction (CSF), spatial masking model (e.g., using techniques describedby Watson in US Published Patent Application No. US 2006-0165311 A1,which is incorporated by reference herein in its entirety), etc. One wayto approach the constraints per embedding location is to combine thedata with the host at embedding locations and then analyze thedifference between the encoded host with the original. The perceptualmodel then specifies whether an adjustment is noticeable based on thedifference between a visibility threshold function computed for anembedding location and the change due to embedding at that location. Theembedder then can change or limit the amount of adjustment per embeddinglocation to satisfy the visibility threshold function. Of course, thereare various ways to compute adjustments that satisfy a visibilitythreshold, with different sequence of operations. See, e.g., our U.S.patent application Ser. Nos. 14/616,686, 14/588,636 and 13/975,919,Patent Application Publication No. US 2010-0150434 A1, and U.S. Pat. No.7,352,878, already incorporated herein.

The Embedder also computes a robustness model. The computing of arobustness model may include computing a detection metric for anembedding location or region of locations. The approach is to model howwell the decoder will be able to recover the data signal at the locationor region. This may include applying one or more decode operations andmeasurements of the decoded signal to determine how strong or reliablethe extracted signal. Reliability and strength may be measured bycomparing the extracted signal with the known data signal. Below, wedetail several decode operations that are candidates for detectionmetrics within the embedder. One example is an extraction filter whichexploits a differential relationship to recover the data signal in thepresence of noise and host signal interference. At this stage ofencoding, the host interference is derivable by applying an extractionfilter to the modulated host. The extraction filter models data signalextraction from the modulated host and assesses whether the differentialrelationship needed to extract the data signal reliably is maintained.If not, the modulation of the host is adjusted so that it is.

Detection metrics may be evaluated such as by measuring signal strengthas a measure of correlation between the modulated host and variable orfixed data components in regions of the host, or measuring strength as ameasure of correlation between output of an extraction filter andvariable or fixed data components. Depending on the strength measure ata location or region, the embedder changes the amount and location ofhost signal alteration to improve the correlation measure. These changesmay be particularly tailored so as to establish relationships of thedata signal within a particular tile, region in a tile or bit cellpattern of the modulated host. To do so, the embedder adjusts bit cellsthat violate the relationship so that the relationship needed to encodea bit (or M-ary symbol) value is satisfied and the thresholds forperceptibility are satisfied. Where robustness constraints are dominant,the embedder will exceed the perceptibility threshold where necessary tosatisfy a desired robustness threshold.

The robustness model may also model distortion expected to be incurredby the modulated host, apply the distortion to the modulated host, andrepeat the above process of measuring detection metrics and adjustingthe amount of alterations so that the data signal will withstand thedistortion. See, e.g., Ser. Nos. 14/616,686, 14/588,636 and 13/975,919for image related processing.

This modulated host is then output as an output image signal 232, with adata channel encoded in it. The operation of combining also may occur inthe analog realm where the data signal is transformed to a renderedform, such as a layer of ink or coating applied by a commercial press tosubstrate. Another example is a data signal that is overprinted as alayer of material, engraved in, or etched onto a substrate, where it maybe mixed with other signals applied to the substrate by similar or othermarking methods. In these cases, the embedder employs a predictive modelof distortion and host signal interference, and adjusts the data signalstrength so that it will be recovered more reliably. The predictivemodeling can be executed by a classifier that classifies types of noisesources or classes of host image and adapts signal strength andconfiguration of the data pattern to be more reliable to the classes ofnoise sources and host image signals that the encoded data signal islikely to be encounter or be combined with.

The output 232 from the Embedder signal typically incurs various formsof distortion through its distribution or use. For printed objects, thisdistortion occurs through rendering an image with the encoded signal inthe printing process, and subsequent scanning back to a digital imagevia a camera or like image sensor.

Turning to FIG. 2, the signal decoder receives an encoded host signal240 and operates on it with one or more processing stages to detect adata signal, synchronize it, and extract data.

The decoder is paired with an input device in which a sensor captures ananalog form of the signal and an analog to digital converter converts itto a digital form for digital signal processing. Though aspects of thedecoder may be implemented as analog components, e.g., such aspreprocessing filters that seek to isolate or amplify the data channelrelative to noise, much of the decoder is implemented as digital signalprocessing modules that implement the signal processing operationswithin a scanner. As noted, these modules can be implemented as softwareinstructions executed within an image scanner or camera, an FPGA, orASIC, etc.

The detector 242 is a signal processing module that detects presence ofthe data channel. The incoming signal is referred to as a suspect hostbecause it may not have a data channel or may be so distorted as torender the data channel undetectable. The detector is in communicationwith a protocol selector 244 to get the protocols it uses to detect thedata channel. It may be configured to detect multiple protocols, eitherby detecting a protocol in the suspect signal and/or inferring theprotocol based on attributes of the host signal or other sensed contextinformation. A portion of the data signal may have the purpose ofindicating the protocol of another portion of the data signal. As such,the detector is shown as providing a protocol indicator signal back tothe protocol selector 244.

The synchronizer module 246 synchronizes the incoming signal to enabledata extraction. Synchronizing includes, for example, determining thedistortion to the host signal and compensating for it. This processprovides the location and arrangement of encoded data elements withinthe host signal.

The data extractor module 248 gets this location and arrangement and thecorresponding protocol and demodulates a data signal from the host. Thelocation and arrangement provide the locations of encoded data elements.The extractor obtains estimates of the encoded data elements andperforms a series of signal decoding operations.

As detailed in examples below and in the incorporated documents, thedetector, synchronizer and data extractor may share common operations,and in some cases may be combined. For example, the detector andsynchronizer may be combined, as initial detection of a portion of thedata signal used for synchronization indicates presence of a candidatedata signal, and determination of the synchronization of that candidatedata signal provides synchronization parameters that enable the dataextractor to apply extraction filters at the correct orientation, scaleand start location of a tile. Similarly, data extraction filters usedwithin data extractor may also be used to detect portions of the datasignal within the detector or synchronizer modules. The decoderarchitecture may be designed with a data flow in which common operationsare re-used iteratively, or may be organized in separate stages inpipelined digital logic circuits so that the host data flows efficientlythrough the pipeline of digital signal operations with minimal need tomove partially processed versions of the host data to and from a sharedmemory unit, such as a RAM memory.

Signal Generator

FIG. 3 is a flow diagram illustrating operations of a signal generator.Each of the blocks in the diagram depict processing modules thattransform the input auxiliary data into a digital payload data signalstructure. The input auxiliary data may include, e.g., a Global TradeItem Number (GTIN) developed by GS1. For example, the GTIN may bestructured in the GTIN-12 format for UPC codes. Of course, the inputauxiliary data may represent other plural bit codes as well. For a givenprotocol, each block provides one or more processing stage optionsselected according to the protocol. In processing module 300, theauxiliary data payload is processed to compute error detection bits,e.g., such as a Cyclic Redundancy Check (CRC), Parity, check sum or likeerror detection message symbols. Additional fixed and variable messagesused in identifying the protocol and facilitating detection, such assynchronization signals may be added at this stage or subsequent stages.

Error correction encoding module 302 transforms the message symbols ofthe digital payload signal into an array of encoded message elements(e.g., binary or M-ary elements) using an error correction method.Examples include block codes, BCH, Reed Solomon, convolutional codes,turbo codes, etc.

Repetition encoding module 304 repeats and concatenates the string ofsymbols from the prior stage to improve robustness. For example, certainmessage symbols may be repeated at the same or different rates bymapping them to multiple locations within a unit area of the datachannel (e.g., one unit area being a tile of bit cells, as describedfurther below).

Repetition encoding may be removed and replaced entirely with errorcorrection coding. For example, rather than applying convolutionalencoding (1/3 rate) followed by repetition (repeat three times), thesetwo can be replaced by convolution encoding to produce a coded payloadwith approximately the same length.

Next, carrier modulation module 306 takes message elements of theprevious stage and modulates them onto corresponding carrier signals.For example, a carrier might be an array of pseudorandom signalelements, with equal number of positive and negative elements (e.g., 16,32, 64 elements), or other waveform. We elaborate further on signalconfigurations below.

Mapping module 308 maps signal elements of each modulated carrier signalto locations within the channel. In the case where a digital host signalis provided, the locations correspond to embedding locations within thehost signal. The embedding locations may be in one or more coordinatesystem domains in which the host signal is represented within a memoryof the signal encoder. The locations may correspond to regions in aspatial domain, temporal domain, frequency domain, or some othertransform domain. Stated another way, the locations may correspond to avector of host signal features, which are modulated to encode a datasignal within the features.

Mapping module 308 also maps a synchronization signal to embeddinglocations within the host signal, for embodiments employing an explicitsynchronization signal. An explicit synchronization signal is describedfurther below.

To accurately recover the payload, the decoder extracts estimates of thecoded bits at the embedding locations within each tile. This requiresthe decoder to synchronize the image under analysis to determine theembedding locations. For images, where the embedding locations arearranged in two dimensional blocks within a tile, the synchronizerdetermines rotation, scale and translation (origin) of each tile. Thismay also involve approximating the geometric distortion of the tile byan affine transformation that maps the embedded signal back to itsoriginal embedding locations.

To facilitate synchronization, the auxiliary signal may include anexplicit or implicit synchronization signal. An explicit synchronizationsignal is an auxiliary signal separate from the encoded payload that isembedded with the encoded payload, e.g., within the same tile). Animplicit synchronization signal is a signal formed with the encodedpayload, giving it structure that facilitates geometric/temporalsynchronization. Examples of explicit and implicit synchronizationsignals are provided in our previously cited U.S. Pat. Nos. 6,614,914,and 5,862,260, which are each hereby incorporated herein by reference intheir entirety.

In particular, one example of an explicit synchronization signal is asignal comprised of a set of sine waves, with pseudo-random phase, whichappear as peaks in the Fourier domain of the suspect signal. See, e.g.,U.S. Pat. Nos. 6,614,914, and 5,862,260, describing use of asynchronization signal in conjunction with a robust data signal. Alsosee U.S. Pat. No. 7,986,807, which is hereby incorporated by referencein its entirety.

Our US Patent Application Publication No. US 2012-0078989 A1, which ishereby incorporated by reference in its entirety, provides additionalmethods for detecting an embedded signal with this type of structure andrecovering rotation, scale and translation from these methods.

Examples of implicit synchronization signals, and their use, areprovided in U.S. Pat. Nos. 6,614,914 and 5,862,260, as well as U.S. Pat.Nos. 6,625,297 and 7,072,490, and U.S. patent application Ser. No.14/724,729 (published as US 2016-0217547 A1), which are herebyincorporated by reference in their entirety.

Signal Embedding in Host

FIG. 4 is a diagram illustrating embedding of an auxiliary signal intohost signal. As shown, the inputs are a host signal block (e.g., blocksof a host digital image) (320) and an encoded auxiliary signal (322),which is to be inserted into the signal block. The encoded auxiliarysignal may include an explicit synchronization component, or the encodedpayload may be formulated to provide an implicit synchronization signal.Processing block 324 is a routine of software instructions or equivalentdigital logic configured to insert the mapped signal(s) into the host byadjusting the corresponding host signal sample(s) at an embeddinglocation according to the value of the mapped signal element. Forexample, the mapped signal is added/subtracted from corresponding asample value, with scale factor and threshold from the perceptual modelor like mask controlling the adjustment amplitude. In implementationswith an explicit synchronization signal, the encoded payload andsynchronization signals may be combined and then added, or addedseparately with separate mask coefficients to control the signalamplitude independently.

Applying the method of FIG. 3, the product or label identifier (e.g., inGTIN format) and additional flag or flags used by control logic areformatted into a binary sequence, which is encoded and mapped to theembedding locations of a tile. For sake of illustration, we describe animplementation of a tile having 256 by 256 embedding locations, wherethe embedding locations correspond to spatial domain embedding locationswithin an image. In particular, the spatial locations correspond topixel samples at a configurable spatial resolution, such as 100 DPI or300 DPI. In this example, we will explain the case where the spatialresolution of the embedded signal is 300 DPI, for an embodiment wherethe resulting image with encode data is printed on a package or labelmaterial, such as a paper, plastic or like substrate. The payload isrepeated in contiguous tiles each comprised of 256 by 256 of embeddinglocations. With these embedding parameters, an instance of the payloadis encoded in each tile, occupying a block of host image of about 1.28by 1.28 inches. These parameters are selected to provide a printedversion of the image on paper or other substrate. At this size, the tilecan be redundantly encoded in several contiguous tiles, providing addedrobustness. An alternative to achieving desired payload capacity is toencode a portion of the payload in smaller tiles, e.g., 128 by 128, anduse a protocol indicator to specify the portion of the payload conveyedin each 128 by 128 tile. Erasure codes may be used to convey differentpayload components per tile and then assemble the components in thedecoder, as discussed in U.S. Pat. No. 9,311,640, which is herebyincorporated herein by reference in its entirety.

Following the construction of the payload, error correction coding isapplied to the binary sequence. This implementation applies aconvolutional coder at rate 1/4, which produces an encoded payloadsignal of 4096 bits. Each of these bits is modulated onto a binaryantipodal, pseudorandom carrier sequence (−1, 1) of length 16, e.g.,multiply or XOR the payload bit with the binary equivalent of chipelements in its carrier to yield 4096 modulated carriers, for a signalcomprising 65,536 elements. These elements map to the 65,536 embeddinglocations in each of the 256 by 256 tiles.

An alternative embodiment, for robust encoding on packaging employstiles of 128 by 128 embedding locations. Through convolutional coding ofan input payload at rate 1/3 and subsequent repetition coding, anencoded payload of 1024 bits is generated. Each of these bits ismodulated onto a similar carrier sequence of length 16, and theresulting 16,384 signal elements are mapped to the 16,384 embeddinglocations within the 128 by 128 tile.

There are several alternatives for mapping functions to map the encodedpayload to embedding locations. In one, these elements have apseudorandom mapping to the embedding locations. In another, they aremapped to bit cell patterns of differentially encoded bit cells asdescribed in U.S. patent application Ser. No. 14/724,729. In the latter,the tile size may be increased to accommodate the differential encodingof each encoded bit in a pattern of differential encoded bit cells,where the bit cells corresponding to embedding locations at a targetresolution (e.g., 300 DPI).

Our U.S. patent application Ser. No. 14/725,399, describes methods forinserting auxiliary signals in areas of package and label designs thathave little host image variability. These methods are particularlyuseful for labels, including price change labels and fresh food labels.These signal encoding methods may be ported to the printing sub-systemin scales used within fresh food, deli and meat departments to encodeGTINs and control flags for variable weight items in the image of alabel, which is then printed by the printer sub-system (typically athermal printer) on the label and affixed to an item.

For an explicit synchronization signal, the mapping function maps adiscrete digital image of the synchronization signal to the host imageblock. For example, where the synchronization signal comprises a set ofFourier magnitude peaks or sinusoids with pseudorandom phase, thesynchronization signal is generated in the spatial domain in a blocksize coextensive with the 256 by 256 tile (or other tile size, e.g., 128by 128) at target embedding resolution.

Various detailed examples of encoding protocols and processing stages ofthese protocols are provided in our prior work, such as our U.S. Pat.Nos. 6,614,914, 5,862,260, and 6,674,876, which are hereby incorporatedby reference, and US Patent Publication No. US 2010-0150434 A1 and U.S.patent application Ser. No. 14/725,399, previously incorporated. Morebackground on signaling protocols, and schemes for managingcompatibility among protocols, are provided in U.S. Pat. No. 7,412,072,which is hereby incorporated by reference.

One signaling approach, which is detailed in U.S. Pat. Nos. 6,614,914,and 5,862,260, is to map elements to pseudo-random locations within achannel defined by a domain of a host signal. See, e.g., FIG. 9 of U.S.Pat. No. 6,614,914. In particular, elements of a watermark signal areassigned to pseudo-random embedding locations within an arrangement ofsub-blocks within a block (referred to as a “tile”). The elements ofthis watermark signal correspond to error correction coded bits. Thesebits are modulated onto a pseudo-random carrier to produce watermarksignal elements (block 306 of FIG. 3), which in turn, are assigned tothe pseudorandom embedding locations within the sub-blocks (block 308 ofFIG. 3). An embedder module modulates this signal onto a host signal byincreasing or decreasing host signal values at these locations for eacherror correction coded bit according to the values of the correspondingelements of the modulated carrier signal for that bit.

FIG. 5 is a flow diagram illustrating a method for decoding a payloadsignal from a host image signal. Implementations of a watermark decoderand watermark processors available from Digimarc Corporation include:

Digimarc Mobile Software Development Kit; and

Digimarc Embedded Systems SDK.

The Embedded Systems SDK is the one typically integrated into scannerhardware.

Corresponding encoder embodiments available from Digimarc Corporationinclude:

Digimarc Barcode SDKs

Digimarc Barcode Plugin

Returning to FIG. 5, the frames are captured at a resolution preferablynear the resolution at which the auxiliary signal has been encodedwithin the original image (e.g., 300 DPI, 100 DPI, etc.). An imageup-sampling or down-sampling operation may be performed to convert theimage frames supplied by the imager to a target resolution for furtherdecoding.

The resulting image blocks supplied to the decoder from these frames maypotentially include an image with the payload. At least some number oftiles of encoded signal may be captured within the field of view, if anobject with encoded data is being scanned. Otherwise, no encoded tileswill be present. The objective, therefore, is to determine asefficiently as possible whether encoded tiles are present.

In the initial processing of the decoding method, it is advantageous toselect frames and blocks within frames that have image content that aremost likely to contain the encoded payload. From the image passed to thedecoder, the decoder selects image blocks for further analysis. Theblock size of these blocks is set large enough to span substantially allof a complete tile of encoded payload signal, and preferably a clusterof neighboring tiles. However, because the distance from the camera mayvary, the spatial scale of the encoded signal is likely to vary from itsscale at the time of encoding. This spatial scale distortion is furtheraddressed in the synchronization process.

For more on block selection, please see co-pending U.S. application Ser.No. 14/332,739, published as US 2015-0030201 A1, and issued as U.S. Pat.No. 9,521,291, which are each hereby incorporated by reference in itsentirety.

Please also see U.S. Provisional Patent Application No. 62/174,454,filed Jun. 11, 2015, and U.S. patent application Ser. No. 15/176,498,filed Jun. 8, 2016 (issued as U.S. Pat. No. 9,922,220), which are herebyincorporated herein by reference in their entirety, for more on blockselection where processing time is limited.

The first stage of the decoding process filters the image to prepare itfor detection and synchronization of the encoded signal (402). Thedecoding process sub-divides the image into blocks and selects blocksfor further decoding operations. For color images, a first filteringstage converts the input color image signal (e.g., RGB values) to acolor channel or channels where the auxiliary signal has been encoded.See, e.g., U.S. Pat. No. 9,117,268, which is hereby incorporated hereinby reference in its entirety, for more on color channel encoding anddecoding. For an image captured under red illumination by a monochromescanner, the decoding process operates on this “red” channel sensed bythe scanner. Some scanners may pulse LEDs of different color to obtainplural color or spectral samples per pixel as described in our PatentApplication Publication No. US 2013-0329006 A1, which is herebyincorporated by reference.

A second filtering operation isolates the auxiliary signal from the hostimage. Pre-filtering is adapted for the auxiliary signal encodingformat, including the type of synchronization employed. For example,where an explicit synchronization signal is used, pre-filtering isadapted to isolate the explicit synchronization signal for thesynchronization process.

In some embodiments, the synchronization signal is a collection of peaksin the Fourier domain. Prior to conversion to the Fourier domain, theimage blocks are pre-filtered. See, e.g., LaPlacian pre-filter in U.S.Pat. No. 6,614,914. A window function is applied to the blocks and thena transform to the Fourier domain, applying an FFT. Another filteringoperation is performed in the Fourier domain. See, e.g., pre-filteringoptions in U.S. Pat. Nos. 6,988,202, 6,614,914, and 9,182,778, which arehereby incorporated by reference in their entirety.

For more on filters, also see U.S. Pat. No. 7,076,082, which is herebyincorporated by reference in its entirety. This patent describes amulti-axis filter, e.g., an oct-axis filter. Oct axis compares adiscrete image sample with eight neighbors to provide a compare value(e.g., +1 for positive difference, −1 or negative difference), and sumsthe compare values. Different arrangements of neighbors and weights maybe applied to shape the filter according to different functions. Anotherfilter variant is a cross shaped filter, in which a sample of interestis compared with an average of horizontal neighbors and verticalneighbors, which are then similarly summed.

Next, synchronization process (404) is executed on a filtered block torecover the rotation, spatial scale, and translation of the encodedsignal tiles. This process may employ a log polar method as detailed inU.S. Pat. No. 6,614,914 or least squares approach of U.S. Pat. No.9,182,778, to recover rotation and scale of a synchronization signalcomprised of peaks in the Fourier domain. To recover translation, thephase correlation method of U.S. Pat. No. 6,614,914 is used, or phaseestimation and phase deviation methods of U.S. Pat. No. 9,182,778 areused.

Alternative methods perform synchronization on an implicitsynchronization signal, e.g., as detailed in Ser. No. 14/724,729(published as 20160217547).

Next, the decoder steps through the embedding locations in a tile,extracting bit estimates from each location (406). This process applies,for each location, the rotation, scale and translation parameters, toextract a bit estimate from each embedding location (406). In particle,as it visits each embedding location in a tile, it transforms it to alocation in the received image based on the affine transform parametersderived in the synchronization, and then samples around each location.It does this process for the embedding location and its neighbors tofeed inputs to an extraction filter (e.g., oct-axis or cross shaped). Abit estimate is extracted at each embedding location using filteringoperations, e.g., oct axis or cross shaped filter (see above), tocompare a sample at embedding locations with neighbors. The output(e.g., 1, −1) of each compare operation is summed to provide an estimatefor an embedding location. Each bit estimate at an embedding locationcorresponds to an element of a modulated carrier signal.

The signal decoder estimates a value of each error correction encodedbit by accumulating the bit estimates from the embedding locations ofthe carrier signal for that bit (408). For instance, in the encoderembodiment above, error correction encoded bits are modulated over acorresponding carrier signal with 16 elements (e.g., multiplied by orXOR with a binary anti-podal signal). A bit value is demodulated fromthe estimates extracted from the corresponding embedding locations ofthese elements. This demodulation operation multiplies the estimate bythe carrier signal sign and adds the result. This demodulation providesa soft estimate for each error correction encoded bit.

These soft estimates are input to an error correction decoder to producethe payload signal (410). For a convolutional encoded payload, a Viterbidecoder is used to produce the payload signal, including the checksum orCRC. For other forms of error correction, a compatible decoder isapplied to reconstruct the payload. Examples include block codes, BCH,Reed Solomon, Turbo codes.

Next, the payload is validated by computing the check sum and comparingwith the decoded checksum bits (412). The check sum matches the one inthe encoder, of course. For the example above, the decoder computes aCRC for a portion of the payload and compares it with the CRC portion inthe payload.

At this stage, the payload is stored in shared memory of the decoderprocess. The recognition unit in which the decoder process residesreturns it to the controller via its interface. This may be accomplishedby various communication schemes, such as IPC, shared memory within aprocess, DMA, etc.

II. Sparse Marks

We refer to one embedding approach as “sparse” marking (or “sparsemarks”) as the data carrying signal is formed as a relatively sparsearray of signal elements, compared to a more continuous array of signalelements. For visual media, the sparse array of elements works well onportions of a host image that are uniform or solid tones or appearlargely blank. With greater sophistication in the signaling, it also iseffective in encoding blank areas around text of a document, label,visual display or package, as our signaling schemes employ robust dataencoding strategies to mitigate impact of interference from the text. Inone embodiment, a sparse mark is comprised of a pattern of spatiallocations where ink is deposited or not. For example, the sparse signalmay be comprised of ink dots on a light background, such that the signalforms a pattern of subtly darker spatial locations. The signal isdesigned to be sparse by the spacing apart of the darker locations onthe light background. Conversely, the signal may be designed as an arrayof lighter “holes” on a relatively darker background. See, for example,U.S. Pat. Nos. 6,345,104, 6,993,152 and 7,340,076, which are herebyincorporated by reference in their entirety.

The sparse signal has minimal impact on visual quality due to its sparsearrangement. However, the trade-off for applications like automaticobject identification is that more sophistication is required in thedata signaling methodology to ensure that the data carried within thesparse signal may be reliably and efficiently recovered in manydifferent and challenging environments. The sparse nature of the signaldictates that less payload may be encoded per unit of object surfacearea. Further, within the sparse signal, there is a trade-off betweenallocating signal for payload capacity versus signal for robustness. Inthe latter category of robustness, the signaling scheme must supportrecovery in environments of geometric distortion, which occurs when thesparse signal is imaged from various angles, perspectives and distances,in the presence of noise of various types that tends to interfere withthe data signal.

There are various sources of geometric distortion that need to beaddressed to reliably recover the payload in the sparse signal. Examplesof geometric distortion include signal cropping and warping. Croppingtruncates portions of the sparse signal, e.g., in cases where only aportion is captured due to occlusion by other objects or incompletecapture by a scanner. Warping occurs when the surface on which thesparse signal is applied is curved (on cups or cans) or wrinkled (onbags and flexible plastic or foil pouches) and when the sparse signal isimaged from a surface at various perspectives.

The design of a signaling scheme must also account for practicalchallenges posed by constraints on digital circuitry, processors andmemory for encoding and decoding. These include computationalefficiency, power consumption, memory consumption, memory bandwidth, useof network bandwidth, cost of hardware circuitry or programmableprocessors/circuitry, cost of designing and integrating encoders anddecoders within signal transmitter and receiver, equipment, etc. Forexample, some encoding schemes may provide optimized encoding ordecoding, but may not be applicable because they are too slow forencoding or decoding in real time, e.g., as the host signal is beingtransmitted, received, updated, or being processed with multiple othersignal processing operations concurrently.

One consideration in the design of a sparse signal is the allocation ofsignal for data carrying and for synchronization. Another considerationis compatibility with other signaling schemes in terms of both encoderand decoder processing flow. With respect to the encoder, the sparseencoder should be compatible with various signaling schemes, includingdense signaling, so that it each signaling scheme may be adaptivelyapplied to different regions of an image design, as represented in animage design file, according to the characteristics of those regions.This adaptive approach enables the user of the encoder tool to selectdifferent methods for different regions and/or the encoder tool to beprogrammed to select automatically a signaling strategy that willprovide the most robust signal, yet maintain the highest quality image,for the different regions.

One example of the advantage of this adaptive approach is in productpackaging where a package design has different regions requiringdifferent encoding strategies. One region may be blank, another blankwith text, another with a graphic in solid tones, another with aparticular spot color, and another with variable image content.

With respect to the decoder, this approach simplifies decoderdeployment, as a common decoder can be deployed that decodes varioustypes of data signals, including both dense and sparse signals.

One approach to sparse signal design is to construct the signal to haveoptimal allocation of payload and synchronization components, withoutregard to compatibility with legacy dense signaling protocols. In suchan approach, the signaling techniques for data and synchronization aredeveloped to minimize interference between the variable data carryingand synchronization functions of the sparse signal. For example, if thesparse signal is being designed without needing to be compatible with adense signaling strategy, it can be designed from the start to becomprised as an array of sparse elements, with variable data and syncfunctions. One advantage is that there is no need to apply a thresholdor quantizer to remove aspects of a signal to convert it into a sparseformat.

Another approach is to design a sparse signal to be compatible with alegacy signaling scheme. Within this type of an approach, one can employtechniques to convert a legacy signaling scheme into a sparse signal. Inparticular, in one such approach, the process of generating a sparsesignal begins with a dense watermark signal, and selectively removeselements of it to produce a sparse signal, while retaining sufficientamounts of data and synchronization functionality.

As we detail further below, there are several ways to convert densesignals to sparse signals. Before exploring these methods, we start byfurther considering properties of dense signals relative to sparsesignal. In some cases, a dense signal is comprised of a multi-valuedwatermark tile (e.g., eight bit per pixel image approximating acontinuous signal), which is a block of m by n embedding locations,where m and n are the integer coordinates of embedding locations in atile (e.g., m=n=128, 256, 512, etc.). The value at each tile correspondsto an adjustment to be made to a corresponding location in a host imageto encode the watermark. The tile is repeated contiguously in horizontaland vertical directions over a region of the host image, possibly theentire image. The signal is considered “dense” relative to a sparsesignal, when the adjustments are densely spaced, in contrast to a sparsesignal, where its signal elements are spread apart in the tile. Densesignals are preferred for host signals that are similarly dense,varying, and multi-valued, enabling embedding by adjusting the values ofthe host signal at the embedding locations. A dense embedding enableshigher capacity embedding for both data and sync functions within atile.

Converting a dense signal to a sparse signal still achieves theobjective of reliable signaling due to a couple of characteristics ofthe signal and host. First, the signal is redundant in the tile andacross repeated tiles, so removing a portion of it from each tile leavessufficient signal for reliable and complete recovery of the payload.Signal detection is aggregated across tiles to further assist inreliable recovery, as detailed, for example in U.S. Pat. No. 6,614,914.Second, sparse signaling is adaptively applied where there is lesslikely to be interference with host signal content, and as such, itssparse property is relatively less impacted by interference.

Some approaches to converting dense to sparse signals include, but arenot limited to:

-   -   Quantizing the array of multi-valued signal to produce a sparse        array of elements by quantizing some sub-set of the values to        zero;    -   Selecting a sub-set of a dense signal, with selection being        adapted to retain data signal and sync function within a tile        (keeping in mind that such selection may be implemented across        tile boundaries in a manner that reliable detection can be made        with the aid of extraction from an area larger than that of a        single tile);    -   Selecting locations to retain based on a particular signal        pattern, which may be variable or fixed per tile;    -   Selection or locations based on a pattern of the data signal or        a synchronization signal; and    -   Combinations of the above, where, for example, quantizing        inherently acts to select values to retain and sets the value of        the sparse element.

These methods are not mutually exclusive and may be combined in variousways. The case of using quantization may also include applying a fixedor adaptive threshold operation to convert a multi-valued dense signalto a sparse signal. Use of a threshold operation to generate a sparsesignal is described, for example, in U.S. Pat. No. 6,993,152, which isincorporated by reference above. Below, we describe further detailsthrough examples illustrating various methods.

Whether one starts with a sparse signal or generates one by converting adense signal, it should be noted that techniques for modulating variabledata into the sparse signal can vary quite a bit. Our U.S. Pat. Nos.5,862,260, 6,614,914, and 6,345,104 describe several examples ofmodulation for carrying variable data in image content, and U.S. patentapplication Ser. No. 14/724,729, which are all hereby incorporatedherein by reference in their entirety, describes yet additionalexamples, including differential modulation methods. These documentsalso describe explicit and implicit synchronization signals.

As introduced above with reference to FIG. 3, there are stages ofmodulation/demodulation in the encoder, so it is instructive to clarifydifferent types of modulation. One stage is where a data symbol ismodulated onto an intermediate carrier signal. Another stage is wherethat modulated carrier is inserted into the host by modulating elementsof the host. In the first case, the carrier might be pattern, e.g., apattern in a spatial domain or a transform domain (e.g., frequencydomain). The carrier may be modulated in amplitude, phase, frequency,etc. The carrier may be, as noted, a pseudorandom string of 1's and 0'sor multi-valued elements that is inverted or not (e.g., XOR, or flippedin sign) to carry a payload or sync symbol.

As noted in our application Ser. No. 14/724,729, carrier signals mayhave structures that facilitate both synchronization and variable datacarrying capacity. Both functions may be encoded by arranging signalelements in a host channel so that the data is encoded in therelationship among signal elements in the host. Application Ser. No.14/724,729 specifically elaborates on a technique for modulating, calleddifferential modulation. In differential modulation, data is modulatedinto the differential relationship among elements of the signal. In somewatermarking implementations, this differential relationship isparticularly advantageous because the differential relationship enablesthe decoder to minimize interference of the host signal by computingdifferences among differentially encoded elements. In sparse signaling,there may be little host interference to begin with, as the host signalmay lack information at the embedding location.

Nevertheless, differential modulation may be exploited or the scheme maybe adapted to allow it to be exploited for sparse signaling. Forexample, sparse elements may be designed such that they have adifferential relationship to other elements, either within the sparsesignal (e.g. the sync component), or within the host signal (e.g.,neighboring background of each sparse element). A sparse element where adot of ink is applied, for example, has a differential relationship withneighbors, where no ink is applied. Data and sync signals may beinterleaved so that they have such differential relationships. A sparsesignal may be encoded differentially relative to a uniform or solidtone, where some sparse elements darken the tone (e.g., darker dots),and others lighten it (e.g., lighter holes).

Differential schemes may further be employed as a preliminary stage togenerate a dense multi-valued signal, which in turn is converted to asparse signal using the above described schemes for conversion. Theencoder then converts this dense signal to a sparse signal, maintainingwhere possible, differential relationships.

Another form of modulating data is through selection of differentcarrier signals to carry distinct data symbols. One such example is aset of frequency domain peaks (e.g., impulses in the Fourier magnitudedomain of the signal) or sine waves. In such an arrangement, each setcarries a message symbol. Variable data is encoded by inserting severalsets of signal components corresponding to the data symbols to beencoded. The decoder extracts the message by correlating with differentcarrier signals or filtering the received signal with filter bankscorresponding to each message carrier to ascertain which sets of messagesymbols are encoded at embedding locations.

Having now illustrated methods to modulate data into the watermark(either dense or sparse), we now turn to the issue of designing forsynchronization. For the sake of explanation, we categorizesynchronization as explicit or implicit. An explicit synchronizationsignal is one where the signal is distinct from a data signal anddesigned to facilitate synchronization. Signals formed from a pattern ofimpulse functions, frequency domain peaks or sine waves is one suchexample. An implicit synchronization signal is one that is inherent inthe structure of the data signal.

An implicit synchronization signal may be formed by arrangement of adata signal. For example, in one encoding protocol, the signal generatorrepeats the pattern of bit cells representing a data element. Wesometimes refer to repetition of a bit cell pattern as “tiling” as itconnotes a contiguous repetition of elemental blocks adjacent to eachother along at least one dimension in a coordinate system of anembedding domain. The repetition of a pattern of data tiles or patternsof data across tiles (e.g., the patterning of bit cells in our U.S. Pat.No. 5,862,260) create structure in a transform domain that forms asynchronization template. For example, redundant patterns can createpeaks in a frequency domain or autocorrelation domain, or some othertransform domain, and those peaks constitute a template forregistration. See, for example, our U.S. Pat. No. 7,152,021, which ishereby incorporated by reference in its entirety.

The concepts of explicit and implicit signaling readily merge as bothtechniques may be included in a design, and ultimately, both provide anexpected signal structure that the signal decoder detects to determinegeometric distortion.

In one arrangement for synchronization, the synchronization signal formsa carrier for variable data. In such arrangement, the synchronizationsignal is modulated with variable data. Examples include sync patternsmodulated with data.

Conversely, in another arrangement, that modulated data signal isarranged to form a synchronization signal. Examples include repetitionof bit cell patterns or tiles.

These techniques may be further exploited in sparse signal designbecause the common structure for carrying a variable payload andsynchronizing in the decoder is retained in the sparse design, whileminimizing interference between the signal components that provide thesefunctions. We have developed techniques in which one signal component isa carrier of the other component, and in these techniques, the processof generating a sparse signal produce a signal that performs bothfunctions.

The variable data and sync components of the sparse signal may be chosenso as to be conveyed through orthogonal vectors. This approach limitsinterference between data carrying elements and sync components. In suchan arrangement, the decoder correlates the received signal with theorthogonal sync component to detect the signal and determine thegeometric distortion. The sync component is then filtered out. Next, thedata carrying elements are sampled, e.g., by correlating with theorthogonal data carrier or filtering with a filter adapted to extractdata elements from the orthogonal data carrier. Signal encoding anddecoding, including decoder strategies employing correlation andfiltering are described in our co-pending application Ser. No.14/724,729, and these strategies may be employed to implement thisapproach for sparse signaling.

Additional examples of explicit and implicit synchronization signals areprovided in our previously cited U.S. Pat. Nos. 6,614,914, and5,862,260. In particular, one example of an explicit synchronizationsignal is a signal comprised of a set of sine waves, with pseudo-randomphase, which appear as peaks in the Fourier domain of the suspectsignal. See, e.g., U.S. Pat. Nos. 6,614,914, and 5,862,260, describinguse of a synchronization signal in conjunction with a robust datasignal. Also see U.S. Pat. No. 7,986,807, which is hereby incorporatedby reference in its entirety.

Our US Publication US 2012-0078989 A1, which is hereby incorporated byreference in its entirety, provides additional methods for detecting anembedded signal with this type of structure and recovering rotation,scale and translation from these methods.

Additional examples of implicit synchronization signals, and their use,are provided in U.S. Pat. Nos. 6,614,914, 5,862,260, and applicationSer. No. 14/724,729 as well as U.S. Pat. Nos. 6,625,297 and 7,072,490,which are hereby incorporated by reference in their entirety.

Returning now to sparse signal design, we now provide detailed examplesof sparse signaling techniques. FIG. 6 is a diagram illustrating anembodiment of a sparse signal generator. The signal generator startswith a tile of two signal components, one carrying variable data 420,and one providing a synchronization function 422. The synchronizationsignal is multi-valued per pixel, and it is passed through a quantizer424 to convert it to a signal with fewer levels per pixel. In itssimplest form, the quantizer converts the multi-valued signal into abinary signal, represented as black and white pixels, by a thresholdoperation. The threshold operation for each pixel within a tile compareseach value with a threshold. For binary signals, elements below thethreshold are shown as black here, while elements above the thresholdare white. As noted, this is simply representative of a modulation stateof an optical property at a sparse element, such as darker or lighterrelative to background, and is not particularly limited to renderingblack and white pixels.

The variable data signal 420 is comprised of elements having one of twovalues (e.g., 1 or 0, A, −A). As explained previously, a payload signalmay be transformed into a robust data signal through one or moremodulation stages, e.g., error correction and modulating the errorcorrection coded signal onto a binary carrier signal, which is theapproach used in this embodiment. This modulated carrier is mapped topixel locations within the tile to form data tile 420.

The signal generator of FIG. 6 produces a sparse signal by selectivelycombining elements of data tile 420 with the quantized synchronizationsignal 422. In the embodiment illustrated here, the signal generatorperforms a matrix operation 428 that selectively retains components ofthe data and synchronization tiles, while producing a sparse signaloutput 430. One particular matrix operation to generate dark sparseelements on a lighter background, as shown here, is to compute a logicalAND operation between corresponding pixel locations within the data andsynchronization tiles, such that pixels that are both black at the samecoordinates in each tile remain black in the output. For other inputs(white AND white, black AND white, or white AND black), the output pixelis white at that coordinate.

In this approach, the black pixels of the message signal are retained atall coordinates in the tile where the synchronization signal also has ablack pixel. This technique distributes sparse message elements within atile according the spatial distribution of the synchronization signal.It ensures that there sufficient signal energy to carry the payloadrobustly, while preserving sufficient signal energy for synchronization.It also ensures that the sync signal does not interfere with the sparsemessage elements. This approach may be reversed in the case where theobjective is to generate a sparse signal with light holes against adarker background, with quantization level set appropriately (see laterillustrations of setting thresholds for holes in dark background).

This approach also demonstrates a signal generation method in which amulti-valued component is effectively merged with a binary component.The multi-valued synchronization tile is a spatial domain representationof synchronization template formed by peaks in the frequency domain. Thebinary valued payload carrying component is redundantly encoded anddistributed over the tile. In particular, modulated carrier elements,with an equal number of binary 0 and 1 values are spread evenly over thespatial locations within a tile.

The principles of the method may be applied to alternative signalcomponent inputs. The sync and data components may both be multi-valuedand selectively quantized to a binary or M-ary form prior to mergingwith a selective combination of the components per tile location.Alternatively, both the sync and data components may be binary valuedand merged with a logic operation. Finally, the data component may bemulti-valued and the sync component binary valued, with the datacomponent being quantized prior to merging with the sync component. Thematrix operation to combine elements at tile coordinates may be adaptedto retain sync and data components that are compatible (e.g.,consistently valued or falling within the same quantization bin). Thisapproach allows the generator to form sparse marks with dark elements onlighter background, lighter elements on darker background, or acombination of lighter and darker sparse elements against a mid-leveltone background.

Quantization level (including threshold) and merging function may be setwith adaptive parameters to bias the sparse signal toward data or syncelements.

FIG. 7 is a diagram illustrating a refinement of a sparse signalgenerator like the one in FIG. 6. In this refinement, the output of thesparse signal generator is further processed to transform the sparsesignal elements. The sparse signal tile output from the generator hasdimensions of m by n, where m and n are integer coordinates. For thesake of illustration, we use the example of m=n=128. In preparation forapplication to an object, the tile coordinates are mapped to coordinatesin a target spatial resolution, which is typically expressed in Dots PerInch (DPI). In FIG. 7, the mapping of a tile coordinate corresponds to a4 by 4 block, which means that the effective DPI of the tile isone-fourth the DPI of the target image resolution. For example, thesparse mark tile may be generated to be 75 DPI for insertion into animage at 300 DPI, which translates to each tile coordinate (called awaxel) being a 4 by 4 block (waxel region) of pixels in the imagecoordinate system at 300 DPI. We refer to the region as the “bump” andratio of target image resolution to waxel resolution as the bump size.

In the refinement of FIG. 7, light and dark waxels (500, 502) of thesparse tile are converted to the higher output resolution. Thisconversion enables additional flexibility in the shaping and location ofeach sparse element. Light elements 500 simply convert to 4×4 regions oflight elements (504) at the waxel coordinates. In this example of darksparse elements on light background, the flexibility is in the selectionof the location of the dark element. In the technique of FIG. 7 thelocation of the dark element is pseudo-randomly selected from among 4locations within the center 2×2 square within the 4×4 pixel region of awaxel. These four alternative locations are depicted in blocks 506, 508,510 and 512. The resulting converted sparse signal output is shown asoutput tile 514. This conversion of the sparse input signal (e.g., at 75DPI) to sparse output image signal at the target resolution (e.g., 300DPI) does the following:

-   -   It makes the sparse signal more sparse;    -   It varies the location of the sparse element per embedding        location so that sparse elements are not consistently falling on        horizontal rows and vertical columns of the tile to make the        sparse signal less visually perceptible;    -   It provides some protection against errors introduced by dot        gain of the printing process. Even with errors in dot size and        location due to dot gain, the resulting sparse element is still        located within the correct tile region.

As we explain further below, this sparse output signal may also beconverted further in the RIP process and as applied when printed ormarked onto an object surface, or rendered for display on a screen orprojected image.

FIGS. 8-10 depict histograms of signal components to help illustrateaspects of sparse signal generation from different types of signals.FIG. 8 is a histogram of a digital watermark signal component, withwaxel values that are at one of two different levels (−1, 1). This is anexample of a histogram of a binary antipodal watermark tile, generatedby modulating symbols onto binary antipodal carriers (e.g., a chippingsequence) to create message chips which are mapped pseudo-randomly intolocations across the tile.

FIG. 9 is a histogram of another digital watermark signal component withmulti-level values. This is an example of a spatial domain conversion ofa sync signal tile formed as frequency domain peaks with pseudorandomphase.

FIG. 10 is a histogram of a combination of the digital watermark signalcomponents of FIGS. 8 and 9, also depicting an example of a thresholdoperation to generate a binary image comprising black and white pixelsfrom an image comprised of multi-valued pixels. In this example, thebinary anti-podal signal elements are multiplied by a scale factor of 10and then added to the multi-valued signal component with thedistribution of FIG. 9. To create a sparse signal of darker dots on alighter background, a threshold operation is applied, for example at thethreshold level of the dashed line. Tile elements with a value below thethreshold are set to dark (“black”) and tile elements with a value abovethe threshold are set to light (“white”). This diagram provides agraphical depiction of the sparse signal generation process, whichretains signal of both data carrying and sync components. The manner inwhich the payload is modulated onto carriers with half positive and halfnegative values ensures that the complete signal can be recovered fromwaxels of negative values or waxels of positive values. Here, for darkon light background, the negatively valued waxels are retained.Additionally, sufficient signal energy of the sync signal is alsoretained.

FIG. 10 is a diagram illustrating another refinement of the sparsesignal generator of FIG. 6. This refinement leverages the sameflexibility discussed in connection with FIG. 7 in establishing thesparse dot in a bump region. In this case, the sparse dot is located inthe bump region where the sync signal level is at its lowest (for darkon light background sparse marks). A similar approach may be used forsparse holes in a darker background, with the sparse hole located wherethe synch signal level is highest within the bump region. Because ofpossible dot gain errors, this approach, like the one in FIG. 7, limitsthe selection of dot location to the center four pixels of the bumpregion.

In this variant of the sparse signal generation, the multi-valued synctile (600) is provided at the resolution of the target image (e.g., 300DPI in the continuing example, where waxels are at resolution of 75DPI). The low point within the center 4×4 region of the waxel is atlocation 602. The signal generator places the sparse dot at thislocation 602, which is one (606) of the four candidate locations, 604,606, 608, 610, selectable by the signal generator. This variant providesmore sync signal strength as the sparse signal is generated based on amore detailed analysis of the sync signal level within the waxel.

FIG. 12 is a diagram illustrating application of a threshold to acontinuous watermark signal, and the resulting output for threedifferent thresholds. The top three boxes 620, 622 and 624, illustratehistograms of a continuous watermark signal, with three differentthreshold settings, shown as the dashed lines. Waxels with values belowthe threshold are set to black (darker pixels), while values above areset to white (lighter pixels). The selection of thresholds at thesethree different settings corresponds to the binary image signals 626,628 and 630 shown below each histogram. These diagrams illustrate howthe thresholds may be adjust to set the sparseness of the output signal.The strongest signal output for the continuous signal is where thethreshold is set to zero.

FIG. 12 illustrates how the thresholding of the continuous watermarksignal component controls the distribution of the sparse signal elementsin the tile. The technique of combining the binary data signal with thecontinuous sync signal with a logical AND operation has the effect ofdistributing the data signal according to the sync signal.

FIG. 13 illustrates a portion of a sparse signal in magnified state toshow dot structure in more detail and set up our explanation of anadditional transformation of the sparse signal. In this particularexample, the image resolution is 300 DPI, and the black squares are 2×2black pixels at the center of the 4×4 waxel region (the “bump” region ofa waxel, where waxels are at 75 DPI). In contrast to the examples ofFIGS. 5 and 9 where a sparse dot is selected from among the 4 pixels ofthe center 2×2 pixels, here all four of the 2×2 pixels are set to black.

FIG. 14 illustrates the sparse signal of FIG. 13, modified to reduce thesignal using a line screen approach. The sparse signal of FIG. 14 isderived from the signal of FIG. 13 by screening back the black dots from100% to 15% with a 175 line screen. This is just one example of how thesparse signal can be made less perceptible by reducing the sparseelements. In this case, the signal is screened back. Another alternativeis to reduce the sparse elements by diluting the ink used to print it(e.g., diluting the ink to create a 15% ink dot).

While we illustrate several examples with black or dark pixels on alight background, the same approach may be applied in different colorinks, including spot colors. Applying the sparse signal with Cyan ink isparticularly effective where the signal is captured with a scanner thatpredominantly captures image signal around a 660 nm wavelength, likemost commercial 1D barcode scanners. The sparse elements may be reducedby screening, diluted ink, or other reduction techniques applied in theRIP and/or at the time of applying the sparse element to a substrate.

The above examples also show sparse signals are constructed fromcontinuous or multivalued signal components and binary signalcomponents. One component is a variable data carrier while another is async signal. The functions of the components may be reversed.Alternatively, both the data and sync components may be continuoussignals that are selectively quantized and combined.

An alternative sparse signal generation process, for example, is aprocess that begins with sync and data components that are peaks in thefrequency domain. The sync peaks are fixed to form a sync template,whereas the data peaks vary in location in frequency coordinatesaccording to data symbols being encoded. These signal components form acontinuous spatial domain signal when the combined peak signals aretransformed to the spatial domain. This continuous signal is thenconverted to a sparse signal with a threshold operation using theabove-explained approach to generate sparse image signals with both dataand sync components. This approach enables the frequency components forsync and data to be selected so as to minimize interference between thetwo components.

In particular, the frequencies may be chosen to be orthogonal carriersignals, with some for sync, some for data, and some for both sync anddata. The carriers may be modulated with variable data, e.g., usingfrequency shifting, phase shifting, etc.

One benefit of the above techniques is that they are compatible withsignal decoders designed for dense watermark signal counterparts to thesparse signal. For details on decoders, including synchronizationmethods, please see our decoders detailed in U.S. Pat. Nos. 6,614,914,5,862,260, and 6,345,104, and synchronization methods in US 2012-0078989A1, which are each hereby incorporated herein in its entirety.Synchronization methods and variable data demodulation operate in asimilar fashion as in dense watermark schemes. However, as noted, theextraction filters may be adapted to be optimized for sparse markextraction.

Binary, multi-valued and continuous watermark signal components may alsobe generated using various techniques describe in our co-pendingapplication Ser. No. 14/724,729, filed May 28, 2015, which is herebyincorporated herein by reference in its entirety, and which describesvarious watermark signal arrangements, differential modulationstrategies, and synchronization approaches. These binary andmulti-valued signal components may then be converted to sparse signalsusing the techniques described in this document. Though the decoding ofsuch sparse signals follows the dense decoding counterparts, we providean example of the processing flow below.

Even further details of our sparse marking technology can be found inassignee's U.S. patent application Ser. No. 14/725,399, filed May 29,2015, and Ser. No. 15/072,884, filed Mar. 17, 2016, which are eachhereby incorporated herein by reference in its entirety.

III. Color Selection and Ink Trapping

Digimarc is currently providing a machine-readable digital watermarkingsolution for retail packaging and other printed objects, often using theterms “Digimarc Barcode for packages” and/or “Digimarc Barcode”. Amongother advantages, digital watermarking facilitates faster and moreefficient checkout relative to traditional 1D barcode checkout. Digimarcrecently publically announced a broad collaboration with GS1, a globalleader in barcode management, to advance retail product identificationthrough digital watermarking. In one Digimarc solution, a digitalwatermark signal can be printed on a retail package through selectivemodulation of the package's inherent design colors, or by addingadditional ink to the package. In many cases the digital watermarksignal is redundantly provided across the package surface, e.g.,arranged in a tile-like pattern. That is, more than one instance of adigital watermark signal can be provided on a package surface. Thisavoids furiously rotating a package in search of a single 1D barcode onthe package surface at checkout, since any package face can be read.

The majority of retail point-of-sale (POS) scanners currently have anarrow-band red LED, roughly with a peak illumination at or around 660nm. We use the terms “at or around 660 nm” to mean a peak illuminationwithin a range of 630 nm-710 nm. One example is a peak at 660 nm.Another example is a peak at 688 nm. Still another is a peak at 645 nm.Of course, many other peaks are intended to be included within thisrange. Additional details regarding red LED scanners can be found, e.g.,in assignee's U.S. Pat. No. 9,380,186, which is hereby incorporatedherein by reference in its entirety. We use the terms “machine-visionwavelength” in the present patent document to mean a color spectralwavelength at or around a wavelength (e.g., at or around 660 nm) atwhich an image capture device (e.g., a red LED scanner or smartphonecamera) operates.

Color Selection

Not all colors are created equal in terms of ability to carry an encodedsignal, particularly when the signal is to be optically captured by atraditional red LED (at or around 660 nm) scanner. Consider a designhaving original artwork that is lightly colored or includes open whitespaces. To a red LED POS scanner, adding a color (e.g., black), which isvisible (or distinguishable) to the red LED scanner, results in largevisibility impact in the lightly colored or white design areas. Incontrast, when considering a design having original artwork that is verydark or black to the red LED POS scanner, adding a color which isvisible to the POS will not be distinguishable by the scanner over thedark original colors. See, e.g., U.S. Pat. No. 9,380,186 at FIG. 4(color drawings are available in the '186 patent file) and Col. 13, line58—Col. 14, line 12. To optimize detectability by a red LED scanner,digital watermark signals are most recognizable in colors which absorblight (or have low reflectance) in the red region of the spectrum (e.g.,capable of being “seen” or “distinguished” by the red LED scanner),while colors with low absorption (or relatively higher reflectance) arenot seen by a traditional retail red LED scanners. So color selection isimportant when designing packages so as to be favorable carriers forencoded signals including digital watermarking.

What is needed is an objective process to characterize a color's abilityto carry or convey an encoded signal, e.g., in difficult environmentssuch as primarily light-colored designs (e.g., including white or openspaces) and primarily dark-colored designs. In other words, we need tofind which colors are suitable to add to artwork (or which colors tomodify) that can be used to carry an encoded signal with a highrobustness per unit visibility. So the following description in thisSection III describes methods, apparatus, packages and systems forcomparing encoded signal visibility and color error to the robustness ofthe encoded signal, which facilitates automated assistance for productpackaging design, reducing cycle time and/or workflow impact, whileimproving visual quality of the product packaging design with morerobust, reliable product identification.

One of our approaches determines total signal robustness per unitvisibility. Remember that we are concerned with a particularmachine-vision wavelength, e.g., at or around 660 nm. Thismachine-vision dependence can be evidenced by a relationship between thereflectance (R) at or around 660 nm of a target or particular color(R660_(100% color)) at full printed color value (e.g., 100% ink) lessthe reflectance (R) of the substrate at or around 660 nm(R660_(substrate)). In fact, encoded signal robustness can be viewed asbeing proportional to, e.g.:Δ660=R660_(100% color) −R660_(substrate)

To be sure, the absolute value of a target or particular color at oraround a machine-vision wavelength isn't needed to predict therobustness of a signal carried by a target or particular color, sincethe difference between the substrate or background color (e.g., white orblue) and the target or particular color used to carry an encoded signalis preferred. For example, with reference to FIG. 20A, a reflectancedifference (or Δ660) between a white substrate and 15% Cyan around 660nm is illustrated. In the illustrated example, and as shown with anencoded signal patch in FIG. 20B, the difference between the whitesubstrate (left lower box) and the 15% Cyan (right lower box),specifically around 660 nm, Δ660, is useful in evaluating colors. Wehave found that sufficient encoded signal reading can be had with a ΔRin the range of 8%-60+%. In the FIGS. 20A and 20B example, thereflectance difference between the substrate (or background) and the 15%Cyan is preferred to be in a range of 8%-60+%, which it is. Morefavorable results are had with a ΔR at or above 10%, e.g., in a range of10%-20% or 10%-60%. We have found even more favorable results with a ΔRat or above 12%, e.g., in a range of 12%-20% or 12%-60%. Although higherΔR values can result in signal visibility concerns or higher colorerrors, a detector will likely see these higher values even morepronouncedly.

Reflectance (R) can be quantitatively measured, e.g., using aspectrophotometer. Suitable devices are sold by, e.g., PCE AmericasInc., with offices at 711 Commerce Way Suite 8, Jupiter, Fla., USA, suchas the Spectrophotometer PCE-CSM 10 or PCE-CSM 8. Otherspectrophotometers are provided by, e.g., Konica Minolta SensingAmericas, Inc., with offices at 101 Williams Drive Ramsey, N.J., USA,such as the Spectrophotometer CM-5, and by, e.g., X-Rite Corp., withoffices at 4300 44th St. SE, Grand Rapids Mich. USA, such as the RM200QCImaging Spectrocolorimeter or X-Rite Pantone subsidiary's PantoneCAPSURE. Of course, other commercially available spectrophotometers canbe used as well.

Once Δ660 is determined, we can establish a proportional relationship ofΔ660 to a visibility error introduced into a design by a target colorcarrying an encoded signal. This visibility error can be classified bytwo (2) components: i) a color error, e.g., a color shift error or acolor match visibility error (Ecm), and ii) encoded signal error, e.g.,a texture error or an encoded signal visibility error (Ewm).

For the color match visibility error we can use:Ecm=((ΔL*)²+(Δa*)²+(Δb*)²)^(1/2)where, ΔL* is the luminance factor or lightness difference between 100%of the ink and the substrate, Δa* represents a relationship between the‘a’ channel color values of the target or particular color (a_(color))and the ‘a’ channel color values of the substrate (a_(sub)) [e.g.,a_(color)−a_(sub)], and Δb* represents a relationship between the ‘b’channel color values of the target or particular color (b_(color)) andthe ‘b’ channel color values of the substrate (b_(sub)) [e.g.,b_(color)−b_(sub)]. For the encoded signal visibility error we can use:

${Ewm} = \left( {\left( {\Delta\; L^{*}} \right)^{2} + \left( \frac{\Delta\; a^{*}}{8} \right)^{2} + \left( \frac{\Delta\; b^{*}}{16} \right)^{2}} \right)^{\frac{1}{2}}$

To even better understand the encoded signal visibility error pleaseconsider an example where an encoded signal, e.g., a sparse mark,includes relatively more signal energy over the spatial resolutionsshown by the gray box in FIG. 15D. If the luminance and chrominanceContrast Sensitivity Functions (CSFs) are integrated over this gray boxregion, the resultant energy ratios approximate the relative weightsthat can be applied to CIE L*(e.g., 1), a* (e.g., 8) and b* (e.g., 16)to determine Ewm. Additional information regarding CSF's can be found,e.g., in F. L. van Nes and M. A. Bouman, ‘Spatial modulation transfer inthe human eye’, Journal of Optical Society of America, 57(3):401-406,March 1967; and G. J. van der Horst and M. A. Bouman, ‘Spatiotemporalchromaticity discrimination’, Journal of Optical Society of America, 59,1969, which are each hereby incorporated herein by reference.

So let's now look at these two errors with respect to a spectraldependency at a given machine-vision wavelength, e.g., at or around 660nm. One such relationship includes, e.g.:

${RWV} = \frac{\Delta 660}{\left( {\left( {\Delta\; L^{*}} \right)^{2} + \left( \frac{\Delta\; a^{*}}{8} \right)^{2} + \left( \frac{\Delta\; b^{*}}{16} \right)^{2}} \right)^{\frac{1}{2}}}$

In the above equation, RWV represents a proportional encoded signalerror or “robustness per unit watermark visibility” at a givenmachine-vision wavelength, ΔL* represents a relationship of lightness ofthe target or particular color (L_(color)) and lightness of theSubstrate (L_(sub)) [e.g., L_(color)−L_(sub)], Δa* represents arelationship between the ‘a’ channel color values of the target orparticular color (a_(color)) and the ‘a’ channel color values of thesubstrate (a_(sub)) [e.g., a_(color)−a_(sub)], and Δb* represents arelationship between the ‘b’ channel color values of the target orparticular color (b_(color)) and the ‘b’ channel color values of thesubstrate (b_(sub)) [e.g., b_(color)−b_(sub)]. For example, thesubstrate may include a white substrate or other colored substrate. Asdiscussed above, the denominator term generally represents an error dueto an encoded signal “texture” or an introduced signal error whenconveyed by the target or particular color. CIE L*, a* and b* can bemeasured values, or previously determined values relative, e.g., to aparticular substrate.

Next we evaluate robustness per unit color match visibility. Thisfeature can be generalized as being proportion to the color error at amachine-vision wavelength, e.g., at or around 660 nm, associated withthe target or particular color. ΔL*, Δa* and Δb*, below, have the samerelationships as in the above RWV relationship. RCV represents oneexample of a proportional color error.

${RCV} = \frac{\Delta 660}{\left( {\left( {\Delta\; L^{*}} \right)^{2} + \left( {\Delta\; a^{*}} \right)^{2} + \left( {\Delta\; b^{*}} \right)^{2}} \right)^{\frac{1}{2}}}$

If color error and encoded signal error have equal importance, then thetotal signal robustness per unit visibility (RPV) can be determined bycombining RWV and RCV, e.g.:RPV=RWV+RCV

In some cases, we weight RWV and RCV to emphasize or deemphasize theencoded signal error or color error, if one is relatively more importantthan the other. For example, RPV=0.7RWV+0.3RCV, or RPV=0.25RWV+0.75RCV.

FIG. 15A shows various colors and their corresponding “Digimarc BarcodeScore” or abbreviated as “DB Score”. In FIG. 15A we evaluated so-calledPantone “C” colors, which refers to colors printed on a coatedsubstrate. (Pantone “U” colors refers to colors printed on an uncoatedsubstrate. Of course, our scoring techniques would apply well to colorsprinted on an uncoated substrate as well.) The ink and substratereflectance values were also measured, but could be substituted frompublished CIELAB values. The DB Score is associated with the color's RPVvalue by normalizing values so that the highest robustness per unitvisibility value is assigned a score of 100, and the next highest RPVscore is assigned a 96, and so on. Of course, other normalizing schemescould be used, e.g., on a 0-1.0 scale or a percentage, etc. In FIG. 15A,colors are sorted from highest to lowest in DB scores. The DB Scorereflects a per-ink measure of encoded signal robustness per unitvisibility. A high DB Score suggests stronger scanner performance withlow visibility relative to a lower DB Score. In more general terms, theDB Score estimates a suitability of an ink (or target color orparticular color) for use in printed packaging applications and otherimage capture scenarios including a specific dominant detection colorchannel (e.g., red LED scanner). For the Δ660 values used in FIG. 15A,we used reflectance at approximately 660 nm. Of course, we could varythis capture wavelength to anywhere at or around 660 nm, or aimed at aparticular capture device (e.g., tuned to a particular LED scanner likea 640 nm or 688 nm peak scanner).

The color commonly referred to as “Pantone 9520 C” was determined tohave the highest DB Score out of the colors tested relative to a PaperWhite C substrate. Measured values and determined RPV and DB Score areshown in Table 1, below:

TABLE 1 Color Values and DB Score for Paper White C and Pantone 9520 DBColor Name CIELAB_L CIELAB_a CIELAB_b RPV Score Paper 95.8 1.11 −5.410.06824 100 White C Pantone 91.54 −6.82 −3.6 9520 C

Of course, instead of using measured values, Pantone 9520 C and othercolors can be estimated by predetermined values. For example, CIELABcolor space values, e.g.: CIELAB 92-9 0 for Pantone 9520 C. (Pantone9520 U can be estimated by the following color values, e.g.: CIELAB92-17 0). Pantone colors and Paper White C can also be estimated orrepresented with standardized color values, e.g., provided in PhotoShopor from PANTONE's own products (e.g., PantoneLive).

FIG. 15B shows color curves for Paper White C and Pantone 9520 C interms of reflectance and wavelength. And, as seen in FIG. 15C, Pantone9520 C is a good substitute for what a more ideal (but hypothetical)embedding color might look like in terms of reflectance. If a sparsemark is placed directly on a white substrate, of the four process colorsCyan (C), Magenta (M), Yellow (Y), and Black (K), only Cyan and Blackwould been seen by a traditional red LED scanner. Both C and K absorbsome light at or around 660 nm. An ideal embedding color would have aspectrum very close to the spectrum of the substrate, yet it wouldabsorb some percentage of light at or around 660 nm. A difference fromthe background color at or around this wavelength is preferable in orderfor it to be seen by a traditional red LED scanner.

One value in terms of visibility associated with selecting a high DBScore color is shown in FIG. 16. Using a color like black to carry anencoded signal (e.g., a sparse mark in FIG. 16) is highly visible to amachine-vision system, e.g., a red LED scanner. However, using black inkover a lightly colored background, e.g., Paper White C, creates a highlyvisible pattern to a human visual system (HVS). In contrast, using acolor determined through the RPV selection process, e.g., Pantone 9520C, for use over a lightly colored background, e.g., Paper White C,creates a robust machine-readable signal at or around 660 nm that isrelatively less visible to the Human Visual System (HVS) compared toblack, cyan and other colors.

The above RPV color selection process is not limited to a certainsubstrate or background color or a particular machine-vision wavelength.In many cases a design will have dark colors, e.g., Pantone Blue 0821 C.A sparse mark or other encoded signal may be provided over the darkcolor. A color can be selected to carry an encoded signal. For example,using the above RPV selection process, Pantone 333 C was selected as afavorable candidate. Measured values and a determined RPV and DB Scoreare shown in Table 2, below:

TABLE 2 Color Values and DB Score for blue background and Pantone 333 CDB Color Name CIELAB_L CIELAB_a CIELAB_b RPV Score Pantone 76.54 −23.5−26.42 0.05574 100 Blue 0821C Pantone 77.06 −45.98 −5.17 333 C

FIG. 17 shows color curves for Pantone Blue 0821 C and Pantone 333 C interms of reflectance and wavelength. Of course, instead of usingmeasured values, color values could be estimated with standardized colorvalues, e.g., provided in PhotoShop or from one of PANTONE's ownproducts, e.g., PANTONE LIVE. (The Table 2 example assumes that theoverprinting ink (the lighter Pantone 333 C) is opaque. The RPV and DBScore determination would need to be modified fortransparent/semi-transparent inks with an ink blend model, e.g., bymultiplying the two inks or color values.) The circled area in FIG. 17shows a reflectance separation between the two colors, ΔR>12%, whichshould be sufficient to distinguish between an encoded signal (carriedby Pantone 333 C) and the background (Pantone Blue 0821C).

Our RPV color selection process can be implemented in a variety of ways.A first approach determines DB Scores for a plurality of backgroundand/or substrate colors using measured or standard color and reflectancevalues. The resulting DB Scores are compiled in a database, memory ortable for access when needed. For example, a designer may look to the DBScores when selecting a color to carry a digital watermark signal orother encoded signal relative to desired background and/or designcolors.

A second approach determines DB Scores in real time, using colorsprovided, e.g., in a design file or by a designer. In this approach, theRPV color selection process can codified in an application plug-in,provided as an application feature or accessed remotely as a cloudservice or server function. A design file may include a variety ofdifferent colors, e.g., 6-10 colors. For background and solid colorsareas, the RPV color selection process can be invoked to select suitablecolors to carry or convey an encoded signal. In some implementations, auser may limit the search of possible colors to those already present inthe design file. For example, the 6-10 design colors can be evaluated,e.g., in terms of their DB Scores relative to an intended background,substrate or solid color area. The one or two highest DB Scores can beevaluated to carry or convey an encoded signal, e.g., such as a sparseor continuous mark. In a related implementation, given a background orsolid color area, prior to running a DB Score, a subset of all possiblecolors is determined. For example, the subset may include those colorswithin a certain color distance from the background or solid color area.“Certain” in this case can be established, e.g., based on “justnoticeable difference” metrics like 0-1.5 ΔE (color error).

The DB Score described above includes a robustness and a visibilitycomponent. The robustness component can be obtained, e.g., from adifference between paper and ink spectral responses at 660 nm. Thevisibility component is based on ink values, e.g., CIELAB values, andcan be tested using, e.g., an Epson Stylus 4900 ink jet proofer.

In another example, a printed fan book can be produced which includes aDB Score together with Pantone colors so that a designer may visualizethe effect of watermarking a spot color. For example, a solid PANTONEspot color is compared to a watermarked version of the spot color.

However, since a physical book with a large range of spot colors wouldbe expensive to produce, we could simulate spot colors using PantoneExtended Color Gamut (ECG) printing. PANTONE ECG has a large gamut whichcovers 90% of PANTONE spot colors while allowing a total maximum inkcoverage of 257% for any given color if required. A Pantone certifiedprinter can be used since accurate CIELAB values are required forfaithful representation of the visibility of the watermarked samples.Also conventional screening will be used for it to be representative ofcommercial printing.

We have found that our DB Score accurately predicts a suitable color(often the relatively best color for a given set of colors) to use for asparse mark, e.g., for white and lightly colored areas of a design. Thevisibility component of the DB Score can be used to quantify sparse markvisibility for a designer. Initial testing shows that the DB Score alsoworks with a single channel continuous tone watermark.

To prove out our DB Score, a psychophysical test was conducted on arange of ink colors to measure subjective visibility. These subjectivetests were conducted on a range of ink colors which were selected tocover a wide range of color match and watermark visibilities. Thecorrelation between the subjective tests and the objective DB was thendetermined. To ensure accurate CIELAB values were being used in theobjective DB Score determination, solid patches of all the test colorswere printed and measured.

A set of twelve (12) human observers were asked to rate their perceptionof image degradation of twelve (12) color sparse mark patch samplesusing a quality ruler. The quality ruler is shown in FIGS. 23A-23D andincludes a sparse mark that increases in signal strength from left (FIG.23A) to right (FIG. 23D). The quality ruler was made with black ink andthe percentages of black ink used in the sparse mark were chosen to haveequal lightness increments, so that the watermark visibility incrementsbetween FIG. 23A (5% black), FIG. 23B (10% black), FIG. 23C (15% black)and FIG. 23D (20% black) are approximately equal. The sparse markspatial pattern remained the same between FIGS. 23A-23D.

The twelve (12) color sparse mark patch samples were shown one at a timeto each of the 12 test observers. An enlarged patch sample isillustrated in FIG. 25. Thumbnails of 10 of the 12 color sparse markpatch samples are shown in FIG. 24, along with a corresponding solidcolor patch for each Pantone color. We assigned lettering (e.g., AR orZN) to identify each of the patches. Two (2) of the twelve (12) Pantonepatch samples in the subjective visibility testing were repeats. SampleBL was the same as EA, and sample ZN was the same as LY. Thus, EA and LYare not shown in FIG. 24. The sparse mark spatial pattern was the samein each of the samples, and each was printed using 100% of itscorresponding color ink. The Pantone colors were chosen to beapproximately equally spaced in visibility across the quality rulersamples. The patch samples and the quality ruler samples were allprinted with an Epson Stylus 4900 on GMG semimatte 250 proof paper usingthe GMG ColorProof RIP. The test observers only saw the sparse markpatch samples, and not the solid color or corresponding Pantone number.

The patch samples were viewed one at a time at a viewing distance ofapproximately 12″. The observers were asked to judge the overallvisibility of each color sparse mark patch sample (FIG. 24) compared tothe visibility of the standard ruler patches (FIGS. 23A-23D). That is,for each color sparse mark patch sample the observers were asked toidentify the closest ruler patch either FIG. 23A, FIG. 23B, FIG. 23C orFIG. 23D. We assigned 3 equal steps between each of the ruler patches,so an observer could chose a direct match with a patch, closer to onepatch vs. the next patch, equally in-between the one patch and the nextpatch, or closer to the next patch vs the one patch. The mean observerscores for the color sparse mark patch samples are plotted in FIG. 26A.The letters A, B, C and D along the vertical axis correspond with FIG.23A, FIG. 23B, FIG. 23C and FIG. 23D, respectively. In general, thecolors on the far right of FIG. 26A are lighter relative to the colorson the far left of the figure.

Please recall that two (2) of the twelve (12) PANTONE color samples inthe subjective visibility testing were repeats. That is, sample BL wasthe same as EA, and sample ZN was the same as LY. Referring again toFIG. 26A, it can be seen that the subjective measurements of BL and EA,and of ZN and LY are very similar. The repeats proved to be anotherquality data point in our subjective testing. That is, the observerswhere consistent in their rankings of BL & EA, and ZN & LY.

After removing the repeats, the correlation of the remaining ten (10)subjective visibility results to corresponding calculated DB Score asdescribed above are shown in FIG. 26B. A large coefficient ofdetermination (r²=0.9643) was seen between the subjective mean and theobjective visibility DB Score. While 12 observers is not a terriblylarge statistical sample set, it is instructive and illustrative of aclose correlation of a DB Score and subjective testing.

The above methods and processes can be automated. For example, a designfile is obtained for a particular package design. The file is unwrappedto discover which colors are included for the design. A DB Score can bedetermined for each of the colors in the file relative to the substrateor background color(s) indicated in the particular package design, withperhaps additional or substitute colors automatically suggested.

In another implementation, one where a design has flexibility, a mostfavorable color can be selected based on DB Scores. For example, if ablue is needed for the design, and design constraints are flexible, ablue color can be selected with a relatively higher DB Score in relationto other blues and the intended substrate, background or solid colorarea. This allows one color to be intentionally selected over anotherbecause of a higher DB Score or RPV value.

Now let's consider a different type of substrate altogether. Some retailitems have a clear plastic substrate that cover or package what's inside(e.g., a package's “fill”). Two opposite examples of fill may include,e.g., a marshmallows (white) and black beans (black). The same clearplastic can result in a different reflectance for overprinted colorsdepending on the fill color. As a bit of background, various clearsubstrates used in flexographic printing have different opacity. Andopaque white ink is often used as a “base ink” (or background ink) whenprinting on clear substrates. Other colors can be printed over the baseink to carry, e.g., an encoded signal or text, designs, etc. The opacityof the white base ink, however, may vary depending on the printingtechnique and ink's components. When a clear plastic substrate orsubstrate+ opaque white are not entirely opaque, the reflectance at oraround 660 nm (as well as other wavelengths) will depend on the color ofthe fill inside the package.

For example, consider FIGS. 21A & 21B. The top curve in FIG. 21Arepresents reflectivity across wavelengths for clear plastic overprintedwith opaque white ink and including a white fill (e.g., marshmallows).The bottom curve represents measurements of reflectivity acrosswavelengths for the same clear plastic overprinted with the same opaquewhite ink, but including a black fill (e.g., black beans). FIG. 21Bshows a visual approximation of printed patches for clear plasticsubstrate with opaque white ink and a white fill (left box) and clearplastic substrate with opaque white ink and a black fill (right box).

If printing an encoded signal (e.g., a sparse mark in cyan ink) on aclear substrate having opaque white, the Δ660 nm will be impacted byboth the opacity of the opaque white and the brightness of the fill orpackaged material. Thus, to establish a reliable RPV or DB Score foroverprinted clear substrates, both the fill and any substrateoverprinting (e.g., opaque white) can be considered.

With reference to FIG. 22A, two Δ660's are shown. The top value (12.5%)is a reflectance difference between: i) a layer including a clearsubstrate/overprinted opaque white ink/over a white fill and ii) Cyanprinted at a 15% value over a layer including a clearsubstrate/overprinted opaque white ink/over a white fill. The bottomvalue (8.4%) is a reflectance difference between: i) a layer includingthe clear substrate/overprinted opaque white ink/over a black fill, andii) Cyan printed at a 15% value over a layer including the clearsubstrate/overprinted opaque white ink/over the black fill.

To further illustrate this example, FIG. 22B approximates a printedclear plastic overprinted with opaque white ink, 15% Cyan and differentfills. The left side of FIG. 22B represents the clear plastic,overprinted with opaque white ink and having a white fill. (On the leftside, the bottom left box approximates the overprinted opaquewhite/clear plastic/white fill and the bottom right box approximate the15% Cyan/overprinted opaque white/clear plastic/white fill.) The rightside of FIG. 22B represents the clear plastic, overprinted with opaquewhite ink and having a black fill. (On the right side, the bottom leftbox approximates the overprinted opaque white/clear plastic/black filland the bottom right box approximate the 15% Cyan/overprinted opaquewhite/clear plastic/black fill.) The Cyan and opaque white ink on theright side (black fill) appear relatively darker than the Cyan andopaque white ink on the left side (white fill).

A resulting difference in reflectance between the white fill and theblack fill could impact an RPV or DB Score prediction, which couldresult in a less desirable encoded signal robustness outcome. (Thiscould lead to choosing a color which yields a relatively larger Δ660 toinsure that embedding will be seen by an image capture device at amachine-vision wavelength.) Substrates with lower opacity have a reducedencoded signal robustness and visibility. Thus, a reflectance differenceattributable to a package's fill can be considered when determiningrobustness and visibility associated with an encoded signal.

As a partial summary, we can evaluate colors in terms of suitability tocarry an encoded signal for any given substrate/background at a givenimage capture wavelength (e.g., a machine-vision wavelength such asfound in a red LED scanner). We can use a metric (e.g., DB Score) tohelp evaluate robustness in terms of reflectance, encoded signal errorand color error. This metric can be helpful in evaluating relativerobustness vs. visibility for a color and substrate/backgroundcombination, at a given machine-vision wavelength. Other factors, e.g.,reflectance differences due to a package's fill, can be considered whenconsidering relative robustness and visibility.

Ink Trapping for Encoded Signals

Ink trapping includes a technique of printing one ink, e.g., a firstspot color, on top of another ink, e.g., a second spot color. Saidanother way, ink trapping includes the overprinting and adhering of oneink over another ink to produce desired secondary or tertiary colors.For example, ink trapping may include the overprinting of CMY to producevarious shades of Red (R), Green (G) and Blue (B), and/or the inktrapping of and with spot colors. (Ink trapping is different thanso-called “image trapping,” which involves compensating for printingplate registration variation, where two adjacent colors butting eachother can be altered to allow for normal registration variances to existwithout degrading a print design.) “Wet trapping” refers to ink that isbeing laid down on another ink that has not yet dried. “Dry trapping”involves an underlying ink that has already dried. One inkcharacteristic involved with ink trapping is so-called “ink tack”. Inktack can be defined as the “stickiness of an ink” or “adhesive” or“adhesion cling” and can be viewed in terms of relative ink tack, e.g.,more adhesion to paper vs. an under or overprinted ink. Typically, highquality process printing requires good ink trapping of the ink colorswith predictable ink trapping to attain the desired colors. As a result,the inks are wet trapped in descending order from highest tack down tolowest tack.

Our research led to a result of using ink trapping to help convey anencoded signal in an otherwise dark ink area. This technique doesn'trequire color screening of spot colors.

The process involves obtaining a spatial domain representation of anencoded signal, e.g., a sparse mark. The spatial domain representationof the sparse mark is used to guide printing of a first ink on asubstrate. That is, the first ink is laid down on the substrate in asparse mark pattern prior to printing of a second ink, e.g., the secondink being a relatively darker ink compared to the first ink, andgenerally this implies that the first ink with have a relatively higherreflectance properties than the second ink. The second ink is thenoverprinted or flooded over the first ink and substrate. This results inareas including just the second ink over the substrates, and other areasincluding the second ink over the first ink which is over the substrate(2^(nd) ink/1^(st) ink/substrate layer). In one embodiment, a darker inkhas a relatively larger tack or adhesion with the substrate relative toits tack with the first ink. This results in less dark ink (in somecases no dark ink) in spatial areas on top of the first ink (see FIG.18B) relative to spatial areas with no first ink. In the areas with thefirst ink, which represents an encoded signal element, less light isabsorbed relative to the areas with just the darker ink. There is aresulting reflection difference between the overprinted first ink areas(more reflectance) and the darker ink only areas (less reflectance). Inktack (or % tack) can be adjusted to provide a reflectance difference(ΔR) between these two areas. We have found that sufficient reading canbe had with a resulting ΔR in the range of 8%-60%. That is, a reflectionof first areas comprising a layer of 2^(nd) ink/1^(st) ink/substrate is8%-60% different than a reflection of second areas comprising a layer ofsecond ink/substrate. More favorable results are had with a ΔR at orabove 12%, e.g., 12%-20% or 12%-60+%. Although higher ΔR values canstart to result in visibility concerns or higher color errors, adetector will likely see these higher values even more pronouncedly.

In the FIG. 18B example, we used Pantone 9520 C as the first printedink, and Opaque Black as the darker ink (second ink). We found thereflectance of a white substrate alone at approximately 660 nm to beabout 85%, the reflectance of Opaque black/9520/substrate areas to beabout 62%, and the reflectance from an Opaque Black/substrate area to beabout 3%. The ΔR between these areas is around 59%, which is aconservative robustness value for encoded signal detection. For example,this reflection difference between encoded signal elements and blackareas can be readily distinguished by a reader analyzing captured imagedata at or around 660 nm. (We provide FIG. 18A as a contrast ofoverprinting opaque black with Pantone 9520 C. A more uniformly blackarea is a result, which would not yield the reflectance properties weseek.)

Consider a case where a sparse mark is first printed on a substrate witha first ink (e.g., Pantone 9520 C) and then flooded over by a darkercolor, a second ink (Opaque Black). A red LED scanner will effectivelysee a sea of black (attributable to the Opaque black/substrate areas)having lighter “holes” or reflective areas for a sparsely encoded signal(e.g., a sparse mark conveyed by the Opaque black/Pantone 9520C/substrate areas). If a detector is configured to recognize darkerareas as signal elements, a captured image can be inverted (light areasbecome darker, and darker areas become lighter) prior to decoding theencoded signal. Alternatively, and prior to printing an encoded signal,the encoded signal is inverted. The inverted signal is used as a patternin printing. The printed pattern is then overprinted with a darkerdesign ink meeting the above ink trapping conditions (e.g., in terms ofachieving a readable ΔR). The printed design is captured by a red LEDscanner, but since the signal itself has already been inverted prior toprinting, the encoded signal can be read directly from the scan data.

As discussed above in our ink selection section, we can optimize theselection of the sparse mark ink and the flood ink to achieve a desiredrobustness per unit visibility.

IV. Operating Environments

The components and operations of the various described embodiments canbe implemented in modules. Notwithstanding any specific discussion ofthe embodiments set forth herein, the term “module” may refer tosoftware, firmware and/or circuitry configured to perform any of themethods, processes, functions or operations described herein. Softwaremay be embodied as a software package, code, instructions, instructionsets or data recorded on non-transitory computer readable storagemediums. Software instructions for implementing the detailedfunctionality can be authored by artisans without undue experimentationfrom the descriptions provided herein, e.g., written in C, C++, MatLab,Visual Basic, Java, Python, Tcl, Perl, Scheme, Ruby, and assembled inexecutable binary files, etc., in conjunction with associated data.Firmware may be embodied as code, instructions or instruction sets ordata that are hard-coded (e.g., nonvolatile) in memory devices. As usedherein, the term “circuitry” may include, for example, singly or in anycombination, hardwired circuitry, programmable circuitry such as one ormore computer processors comprising one or more individual instructionprocessing cores, parallel processors, state machine circuitry, orfirmware that stores instructions executed by programmable circuitry.

Applicant's work also includes taking the scientific principles andnatural laws on which the present technology rests, and tying them downin particularly defined implementations. One such implementation iselectronic circuitry that has been custom-designed and manufactured toperform some or all of the component acts, as an application specificintegrated circuit (ASIC).

To realize such an implementation, some or all of the technology isfirst implemented using a general purpose computer, using software suchas MatLab (from Mathworks, Inc.). For example, the RPV methods discussedabove can be coded in Matlab, including a determination of RPW and RPC.A tool such as HDLCoder (also available from MathWorks) is next employedto convert the MatLab model to VHDL (an IEEE standard, and doubtless themost common hardware design language). The VHDL output is then appliedto a hardware synthesis program, such as Design Compiler by Synopsis,HDL Designer by Mentor Graphics, or Encounter RTL Compiler by CadenceDesign Systems. The hardware synthesis program provides output dataspecifying a particular array of electronic logic gates that willrealize the technology in hardware form, as a special-purpose machinededicated to such purpose. This output data is then provided to asemiconductor fabrication contractor, which uses it to produce thecustomized silicon part. (Suitable contractors include TSMC, GlobalFoundries, and ON Semiconductors.)

Another specific implementation of the present disclosure includes inkselection process, e.g., RPV/DB Score methods including a determinationof RPW and RPC, operating on a specifically configured smartphone (e.g.,iPhone 6 or Android device) or other mobile device, such phone ordevice. The smartphone or mobile device may be configured and controlledby software (e.g., an App or operating system) resident on thesmartphone device. The resident software may include, e.g., a barcodedecoder, digital watermark detector and detectability measure generatormodule.

For the sake of further illustration, FIG. 19 is a diagram of anelectronic device (e.g., a smartphone, mobile device, tablet, laptop, orother electronic device) in which the components of the above encoder,decoder, and/or various ink or color selection embodiments may beimplemented. It is not intended to be limiting, as the embodiments maybe implemented in other device architectures or electronic circuitry.

Referring to FIG. 19, a system for an electronic device includes bus100, to which many devices, modules, etc., (each of which may begenerically referred as a “component”) are communicatively coupled. Thebus 100 may combine the functionality of a direct memory access (DMA)bus and a programmed input/output (PIO) bus. In other words, the bus 100may facilitate both DMA transfers and direct CPU read and writeinstructions. In one embodiment, the bus 100 is one of the AdvancedMicrocontroller Bus Architecture (AMBA) compliant data buses. AlthoughFIG. 19 illustrates an embodiment in which all components arecommunicatively coupled to the bus 100, it will be appreciated that oneor more sub-sets of the components may be communicatively coupled to aseparate bus in any suitable or beneficial manner, and that anycomponent may be communicatively coupled to two or more buses in anysuitable or beneficial manner. Although not illustrated, the electronicdevice can optionally include one or more bus controllers (e.g., a DMAcontroller, an I2C bus controller, or the like or any combinationthereof), through which data can be routed between certain of thecomponents.

The electronic device also includes a CPU 102. The CPU 102 may be anymicroprocessor, multi-core microprocessor, parallel processors, mobileapplication processor, etc., known in the art (e.g., a ReducedInstruction Set Computer (RISC) from ARM Limited, the Krait CPUproduct-family, any X86-based microprocessor available from the IntelCorporation including those in the Pentium, Xeon, Itanium, Celeron,Atom, Core i-series product families, etc.). Another CPU example is anApple A8 or A7. The A8 is built on a 64-bit architecture, includes amotion co-processor and is manufactured on a 20 nm process. The CPU 102runs an operating system of the electronic device, runs applicationprograms (e.g., mobile apps such as those available through applicationdistribution platforms such as the Apple App Store, Google Play, etc.,or custom designed to include watermark detection and objectauthentication) and, optionally, manages the various functions of theelectronic device. The CPU 102 may include or be coupled to a read-onlymemory (ROM) (not shown), which may hold an operating system (e.g., a“high-level” operating system, a “real-time” operating system, a mobileoperating system, or the like or any combination thereof) or otherdevice firmware that runs on the electronic device. Watermark detectioncapabilities can be integrated into the operating system itself.

The electronic device may also include a volatile memory 104electrically coupled to bus 100. The volatile memory 104 may include,for example, any type of random access memory (RAM). Although not shown,the electronic device may further include a memory controller thatcontrols the flow of data to and from the volatile memory 104.

The electronic device may also include a storage memory 106 connected tothe bus. The storage memory 106 typically includes one or morenon-volatile semiconductor memory devices such as ROM, EPROM and EEPROM,NOR or NAND flash memory, or the like or any combination thereof, andmay also include any kind of electronic storage device, such as, forexample, magnetic or optical disks. In embodiments of the presentinvention, the storage memory 106 is used to store one or more items ofsoftware. Software can include system software, application software,middleware (e.g., Data Distribution Service (DDS) for Real Time Systems,MER, etc.), one or more computer files (e.g., one or more data files,configuration files, library files, archive files, etc.), one or moresoftware components, or the like or any stack or other combinationthereof. Examples of system software include operating systems (e.g.,including one or more high-level operating systems, real-time operatingsystems, mobile operating systems, or the like or any combinationthereof), one or more kernels, one or more device drivers, firmware, oneor more utility programs (e.g., that help to analyze, configure,optimize, maintain, etc., one or more components of the electronicdevice), and the like.

Application software typically includes any application program thathelps users solve problems, perform tasks, render media content,retrieve (or access, present, traverse, query, create, organize, etc.)information or information resources on a network (e.g., the World WideWeb), a web server, a file system, a database, etc. Examples of softwarecomponents include device drivers, software CODECs, message queues ormailboxes, databases, etc. A software component can also include anyother data or parameter to be provided to application software, a webapplication, or the like or any combination thereof. Examples of datafiles include image files, text files, audio files, video files, hapticsignature files, and the like.

Also connected to the bus 100 is a user interface module 108. The userinterface module 108 is configured to facilitate user control of theelectronic device. Thus the user interface module 108 may becommunicatively coupled to one or more user input devices 110. A userinput device 110 can, for example, include a button, knob, touch screen,trackball, mouse, microphone (e.g., an electret microphone, a MEMSmicrophone, or the like or any combination thereof), an IR orultrasound-emitting stylus, an ultrasound emitter (e.g., to detect usergestures, etc.), one or more structured light emitters (e.g., to projectstructured IR light to detect user gestures, etc.), one or moreultrasonic transducers, or the like or any combination thereof.

The user interface module 108 may also be configured to indicate, to theuser, the effect of the user's control of the electronic device, or anyother information related to an operation being performed by theelectronic device or function otherwise supported by the electronicdevice. Thus the user interface module 108 may also be communicativelycoupled to one or more user output devices 112. A user output device 112can, for example, include a display (e.g., a liquid crystal display(LCD), a light emitting diode (LED) display, an active-matrix organiclight-emitting diode (AMOLED) display, an e-ink display, etc.), a light,an illumination source such as a flash or torch, a buzzer, a hapticactuator, a loud speaker, or the like or any combination thereof. In thecase of an iPhone 6, the flash includes a True Tone flash including adual-color or dual-temperature flash that has each color firing atvarying intensities based on a scene to make sure colors and skin tonestay true.

Generally, the user input devices 110 and user output devices 112 are anintegral part of the electronic device; however, in alternateembodiments, any user input device 110 (e.g., a microphone, etc.) oruser output device 112 (e.g., a loud speaker, haptic actuator, light,display, or printer) may be a physically separate device that iscommunicatively coupled to the electronic device (e.g., via acommunications module 114). A printer encompasses many different devicesfor applying our encoded signals to objects, such as 2D and 3D printers,etching, engraving, flexo-printing, offset printing, embossing, lasermarking, etc. The printer may also include a digital press such as HP'sindigo press. An encoded object may include, e.g., a consumer packagedproduct, a label, a sticker, a logo, a driver's license, a passport orother identification document, etc.

Although the user interface module 108 is illustrated as an individualcomponent, it will be appreciated that the user interface module 108 (orportions thereof) may be functionally integrated into one or more othercomponents of the electronic device (e.g., the CPU 102, the sensorinterface module 130, etc.).

Also connected to the bus 100 is an image signal processor 116 and agraphics processing unit (GPU) 118. The image signal processor (ISP) 116is configured to process imagery (including still-frame imagery, videoimagery, or the like or any combination thereof) captured by one or morecameras 120, or by any other image sensors, thereby generating imagedata. General functions typically performed by the ISP 116 can includeBayer transformation, demosaicing, noise reduction, image sharpening,filtering, or the like or any combination thereof. The GPU 118 can beconfigured to process the image data generated by the ISP 116, therebygenerating processed image data. General functions typically performedby the GPU 118 include compressing image data (e.g., into a JPEG format,an MPEG format, or the like or any combination thereof), creatinglighting effects, rendering 3D graphics, texture mapping, calculatinggeometric transformations (e.g., rotation, translation, etc.) intodifferent coordinate systems, etc. and send the compressed video data toother components of the electronic device (e.g., the volatile memory104) via bus 100. The GPU 118 may also be configured to perform one ormore video decompression or decoding processes. Image data generated bythe ISP 116 or processed image data generated by the GPU 118 may beaccessed by the user interface module 108, where it is converted intoone or more suitable signals that may be sent to a user output device112 such as a display, printer or speaker. GPU 118 may also beconfigured to serve one or more functions of a watermark detector. Insome cases GPU 118 searches for a watermark orientation component, whilepayload resolution is performed by the CPU 102.

Also coupled the bus 100 is an audio I/O module 122, which is configuredto encode, decode and route data to and from one or more microphone(s)124 (any of which may be considered a user input device 110) and loudspeaker(s) 126 (any of which may be considered a user output device110). For example, sound can be present within an ambient, auralenvironment (e.g., as one or more propagating sound waves) surroundingthe electronic device. A sample of such ambient sound can be obtained bysensing the propagating sound wave(s) using one or more microphones 124,and the microphone(s) 124 then convert the sensed sound into one or morecorresponding analog audio signals (typically, electrical signals),thereby capturing the sensed sound. The signal(s) generated by themicrophone(s) 124 can then be processed by the audio I/O module 122(e.g., to convert the analog audio signals into digital audio signals)and thereafter output the resultant digital audio signals (e.g., to anaudio digital signal processor (DSP) such as audio DSP 128, to anothermodule such as a song recognition module, a speech recognition module, avoice recognition module, etc., to the volatile memory 104, the storagememory 106, or the like or any combination thereof). The audio I/Omodule 122 can also receive digital audio signals from the audio DSP128, convert each received digital audio signal into one or morecorresponding analog audio signals and send the analog audio signals toone or more loudspeakers 126. In one embodiment, the audio I/O module122 includes two communication channels (e.g., so that the audio I/Omodule 122 can transmit generated audio data and receive audio datasimultaneously).

The audio DSP 128 performs various processing of digital audio signalsgenerated by the audio I/O module 122, such as compression,decompression, equalization, mixing of audio from different sources,etc., and thereafter output the processed digital audio signals (e.g.,to the audio I/O module 122, to another module such as a songrecognition module, a speech recognition module, a voice recognitionmodule, etc., to the volatile memory 104, the storage memory 106, or thelike or any combination thereof). Generally, the audio DSP 128 mayinclude one or more microprocessors, digital signal processors or othermicrocontrollers, programmable logic devices, or the like or anycombination thereof. The audio DSP 128 may also optionally include cacheor other local memory device (e.g., volatile memory, non-volatile memoryor a combination thereof), DMA channels, one or more input buffers, oneor more output buffers, and any other component facilitating thefunctions it supports (e.g., as described below). In one embodiment, theaudio DSP 128 includes a core processor (e.g., an ARM® AudioDE™processor, a Hexagon processor (e.g., QDSP6V5A)), as well as a datamemory, program memory, DMA channels, one or more input buffers, one ormore output buffers, etc. Although the audio I/O module 122 and theaudio DSP 128 are illustrated as separate components, it will beappreciated that the audio I/O module 122 and the audio DSP 128 can befunctionally integrated together. Further, it will be appreciated thatthe audio DSP 128 and other components such as the user interface module108 may be (at least partially) functionally integrated together.

The aforementioned communications module 114 includes circuitry,antennas, sensors, and any other suitable or desired technology thatfacilitates transmitting or receiving data (e.g., within a network)through one or more wired links (e.g., via Ethernet, USB, FireWire,etc.), or one or more wireless links (e.g., configured according to anystandard or otherwise desired or suitable wireless protocols ortechniques such as Bluetooth, Bluetooth Low Energy, WiFi, WiMAX, GSM,CDMA, EDGE, cellular 3G or LTE, Li-Fi (e.g., for IR- or visible-lightcommunication), sonic or ultrasonic communication, etc.), or the like orany combination thereof. In one embodiment, the communications module114 may include one or more microprocessors, digital signal processorsor other microcontrollers, programmable logic devices, or the like orany combination thereof. Optionally, the communications module 114includes cache or other local memory device (e.g., volatile memory,non-volatile memory or a combination thereof), DMA channels, one or moreinput buffers, one or more output buffers, or the like or anycombination thereof. In one embodiment, the communications module 114includes a baseband processor (e.g., that performs signal processing andimplements real-time radio transmission operations for the electronicdevice).

Also connected to the bus 100 is a sensor interface module 130communicatively coupled to one or more sensor(s) 132. Sensor 132 can,for example, include an accelerometer (e.g., for sensing acceleration,orientation, vibration, etc.), a magnetometer (e.g., for sensing thedirection of a magnetic field), a gyroscope (e.g., for trackingrotation, orientation, or twist), a barometer (e.g., for sensing airpressure, from which relative elevation can be determined), a windmeter, a moisture sensor, an ambient light sensor, an IR or UV sensor orother photodetector, a pressure sensor, a temperature sensor, anacoustic vector sensor (e.g., for sensing particle velocity), a galvanicskin response (GSR) sensor, an ultrasonic sensor, a location sensor(e.g., a GPS receiver module, etc.), a gas or other chemical sensor, orthe like or any combination thereof. Although separately illustrated inFIG. 19, any camera 120 or microphone 124 can also be considered asensor 132. Generally, a sensor 132 generates one or more signals(typically, electrical signals) in the presence of some sort of stimulus(e.g., light, sound, moisture, gravitational field, magnetic field,electric field, etc.), in response to a change in applied stimulus, orthe like or any combination thereof. In one embodiment, all sensors 132coupled to the sensor interface module 130 are an integral part of theelectronic device; however, in alternate embodiments, one or more of thesensors may be physically separate devices communicatively coupled tothe electronic device (e.g., via the communications module 114). To theextent that any sensor 132 can function to sense user input, then suchsensor 132 can also be considered a user input device 110. The sensorinterface module 130 is configured to activate, deactivate or otherwisecontrol an operation (e.g., sampling rate, sampling range, etc.) of oneor more sensors 132 (e.g., in accordance with instructions storedinternally, or externally in volatile memory 104 or storage memory 106,ROM, etc., in accordance with commands issued by one or more componentssuch as the CPU 102, the user interface module 108, the audio DSP 128,the cue detection module 134, or the like or any combination thereof).In one embodiment, sensor interface module 130 can encode, decode,sample, filter or otherwise process signals generated by one or more ofthe sensors 132. In one example, the sensor interface module 130 canintegrate signals generated by multiple sensors 132 and optionallyprocess the integrated signal(s). Signals can be routed from the sensorinterface module 130 to one or more of the aforementioned components ofthe electronic device (e.g., via the bus 100). In another embodiment,however, any signal generated by a sensor 132 can be routed (e.g., tothe CPU 102), the before being processed.

Generally, the sensor interface module 130 may include one or moremicroprocessors, digital signal processors or other microcontrollers,programmable logic devices, or the like or any combination thereof. Thesensor interface module 130 may also optionally include cache or otherlocal memory device (e.g., volatile memory, non-volatile memory or acombination thereof), DMA channels, one or more input buffers, one ormore output buffers, and any other component facilitating the functionsit supports (e.g., as described above). In one embodiment, the sensorinterface module 130 may be provided as the “Sensor Core” (SensorsProcessor Subsystem (SPS)) from Qualcomm, the “frizz” from Megachips, orthe like or any combination thereof. Although the sensor interfacemodule 130 is illustrated as an individual component, it will beappreciated that the sensor interface module 130 (or portions thereof)may be functionally integrated into one or more other components (e.g.,the CPU 102, the communications module 114, the audio I/O module 122,the audio DSP 128, the cue detection module 134, or the like or anycombination thereof).

CONCLUDING REMARKS

Having described and illustrated the principles of the technology withreference to specific implementations, it will be recognized that thetechnology can be implemented in many other, different, forms. Toprovide a comprehensive disclosure without unduly lengthening thespecification, applicant hereby incorporates by reference each of theabove referenced patent documents in its entirety. Such patent documentsare incorporated in their entireties, including all drawings andappendices, even if cited above in connection with specific of theirteachings. These documents disclose technologies and teachings that canbe incorporated into the arrangements detailed, and into which thetechnologies and teachings detailed herein can be incorporated.

The particular combinations of elements and features in theabove-detailed embodiments are exemplary only; the interchanging andsubstitution of these teachings with other teachings in this and theincorporated-by-reference patents are also contemplated.

Many combinations will be evident from the above disclosure.

What is claimed is:
 1. A method of providing a design for a printedobject comprising: obtaining information associated with a firstsubstrate comprising a first area, the first area comprising a firstregion and a second region; providing an encoded signal arranged in asparse mark pattern to be printed within the second region with a firstink so that the second region comprises a top layer of ink flood using asecond ink, a middle layer of sparse mark pattern and a bottom layer offirst substrate, in which the first region is to be printed to comprisea top layer of ink flood using the second ink and a layer of firstsubstrate; obtaining information associated with the second ink to beprinted over the sparse mark pattern and the first substrate within thesecond region, in which the second ink comprises a greater tack oradhesion with the first substrate relative to the second ink's tack oradhesion with the first ink; determining whether the first region andthe second region comprise a spectral reflectance difference at amachine-vision wavelength in the range of 8%-60%; and selecting adifferent combination of first ink and second ink when the spectralreflectance difference at a machine-vision wavelength is not within therange of 8%-60%.
 2. The method of claim 1 in which the range comprises12%-20%.
 3. The method of claim 1 in which the spectral reflectancedifference comprises 12% or higher within a range of 12%-60%.
 4. Themethod of claim 1 in which the first region comprises a darker regionrelative to the second region.
 5. The method of claim 1 in which firstreflectance values for the first ink at the machine-vision wavelengthaccount for a package's fill.
 6. The method of claim 5 in which thefirst ink is to be overprinted on a clear plastic substrate.
 7. Themethod of claim 6 in which second reflectance values for the substrateor the second ink at the machine-vision wavelength account for packagefill.
 8. The method of claim 7 in which first substrate comprises theclear plastic substrate.
 9. The method of claim 1 in which the printedobject comprises a product hag tag.
 10. The method of claim 1 in whichthe printed object comprises a product label.
 11. The method of claim 1in which the sparse mark pattern comprises a 2-dimensional patterncomprising a payload component and a synchronization component, in whichthe sparse mark pattern retains payload component elements andsynchronization component elements that are compatible at the samespatial coordinates as one another within the 2-dimensional pattern. 12.A printed object comprising: a first substrate comprising a first area,the first area comprising a first region and a second region, in whichthe second region comprises an encoded signal arranged in a printed2-dimensional pattern, the encoded signal comprising a payload componentand a synchronization component, in which the encoded signal retainspayload component elements and synchronization component elements thatare compatible at the same spatial coordinates as one another within the2-dimensional pattern, the 2-dimensional pattern printed with a firstink so that the second region comprises a top layer of ink flood using asecond ink, a middle layer of printed 2-dimensional pattern and a bottomlayer of first substrate, and in which the first region comprises aprinted top layer of ink flood using the second ink and a layer of firstsubstrate, in which the second ink comprises a greater tack or adhesionwith the first substrate relative to the second ink's tack or adhesionwith the first ink, and in which the first region and the second regioncomprise a spectral reflectance difference at a machine-visionwavelength in the range of 8%-60%.
 13. The printed object of claim 12 inwhich the range comprises 12%-20%.
 14. The printed object of claim 12 inwhich the spectral reflectance difference comprises 12% or higher withina range of 12%-60%.
 15. The printed object of claim 12 in which thefirst region comprises a darker region relative to the second region.16. The printed object of claim 12 in which the first substratecomprises a clear plastic substrate.
 17. The printed object of claim 16in which the clear plastic substrate comprises packaging surrounding adark fill.
 18. The printed object of claim 16 in which the clear plasticsubstrate comprises packaging surrounding a light fill.
 19. The printedobject of claim 12 in which the first substrate comprises productpackaging.